extend ndarray in-place reshape

Description
extend ndarray in-place reshape
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[x] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[x] add MXNDArrayReshape64 which supports extended reshape
[x] use new api in arr.reshape

Doc can be found at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-10360/6/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.reshape


Question: marks vs markers?

Hey, This is not an issue but more of a question regarding the design. What is the difference between markers and marks. I know that markers come from numbers (converted to their shift value) and mars are letters but other than that?
And also there is a limited number of marks + markers available. Why not assign a number to them and keep a list internally. And also make all of them global (or project base)? I think this would have resulted in a simpler design.

Marks are a feature built into vim - lowercase marks are local to the buffer while the uppercase are global and shared across all buffers/files. Thus most decisions regarding marks are dictated by vim's design decisions and not mine. The only thing I added to it was displaying it on the side.
I felt marks were a tad limited in what they could do and wanted to fill in a few holes - one thing in particular was I wanted to group a few of them together and have multiple such groups. Marks are not really suitable for this as they're specific to the line they're used on and as such you can have only 1 type of a mark in a given file. That's where markers come in - you can place multiple markers of the same type and jump between markers of the same/different types. I think that's about the biggest difference between the two.

And also there is a limited number of marks + markers available.

As I mentioned above, the no. of marks available are dictated by how vim implements them and I have no control over it. Similarly, markers were originally meant to expand upon marks and thus were heavily influenced by them. You can see this in how markers are placed/removed, it's very similar to marks i.e. it uses only a single character. Furthermore,
there is the issue of using and representing them - all alphabetical characters were used by marks, special character marks have a specific meaning and are already defined by vim so that left just numeric characters - hence there are only 10 markers by default - 1 symbol per digit. At the end of the day, the way I was intending to use them (in groups), a handful felt sufficient and I didn't push further.

Why not assign a number to them and keep a list internally. And also make all of them global (or project base)? I think this would have resulted in a simpler design.

What you're describing sounds like a way to generically implement bookmarks and if that was the intention here then I'd agree with your statement. However, I don't think of marks and/or markers as true bookmarks; the global marks come close to approximating them but they still fall short in many areas and IMO default vim doesn't support true bookmarking.
Having said that, I do use a number to represent each marker (it's a function of the buffer no. and line no.) and store it in a list. The representation of the marker is decoupled from the storage and I think theoretically there really is no limit to how many markers can be defined. The limits are dictated more by the mechanism of placing/removing them as described above.
Hope this answers all your questions.


Enterprise security support over NetXSecure

Hi,
We are trying to add enterprise security support with NetXDuo network stack and NetXSecure TLS stack and facing following issue.
With enterprise security, TLS handshake needs to be performed with RADIUS server using EAP packets during WiFi connection. NetXSecure API user(enterprise security code) need to get the control over the send and receive functions during the TLS handshake, so that he can create EAP packets over the TLS handshake messages. But currently NetXSecure TLS APIs expects a connected socket from the user and sends the TLS handshake message using the socket pointer. So the enterprise security code does not get control over send and receive functions and cannot create EAP packets for TLS handshake messages. Also during the WiFi connection it is not possible to provide a connected socket to TLS APIs, as the sockets come into picture after the WiFi connection.
Can you please let us know if there is a mechanism to add Enterprise security support with NetXSecure?
Also if there is some implementation of Enterprise security over NetXSecure please point me to that code.
Thanks,
Jayasankar.

Hi @cy-jayasankar,
I am facing a similar issue.
Did you find a way to add EAP to NetX?
Did you switch to another TCP/IP stack to handle it ?
Thanks

Did you have a look at https://github.com/Infineon/nx-secure-wpa3 ?


chore(release): Add release notes for RSS-468

Overview
Add a release note for RSS-468 / PR #14512.
Review requests

Can this be made less wordy?
Do we say "OT-2" or "The OT-2" or "OT-2 robots" these days?
Is the Capitalization of "Labware Position Check" and "Tip Length Calibration" Appropriate?

Risk assessment
No risk.

remaining question that line cited above ☝ is in the app-shell notes. i think that was because of the focus on "hey, re-run LPC, an app feature." this is a pretty similar fix / required user action. so it should probably be moved or copied to the other notes file as well.

I probably wouldn't have added the Thermocycler thing to the app-shell notes. Taking that for granted, though, it sounds like both notes should be in both places, so I've copied them.
Another vote for merging the release note files, I guess.


Fix bug: cancel_label does not work

Bug description:
Originally when I click the "Cancel the label" menu, the Curve frontend will send a request http://localhost:8080/v1/data/live_data?startTime=1528030800000&endTime=1528063200000&action=cancel_label. But it does not actually work and the server does not clear the label. This PR fixes the bug.

Hello, the problem is already being checked and fixed, and it will be online as soon as possible, thank you for your feedback.


Allow custom pull request filters

On GitHub, we have the ability to filter pull requests, and reuse the same filter by storing the URL in a bookmark.

Allow adding our own categories here :point_down: with custom filters using the pre-existing GitHub search qualifiers.

More info on the search qualifiers

I love this idea! I made a prototype of how this could work:

On GitHub.com, it's nice to be able to use the dropdowns to help construct the query. I'm not sure what we could do here to help with query writing, validation seems challenging since my understanding of the query language is that anything not interpreted as a search qualifier would just be used as a keyword.
Other things we could have:

"Edit", which allows you to pick a category, first modify the title, then modify the query
Be able to view the query in the tree?
Have an inline action on a category to either view or modify the query


How to handle activity lifecycle events, specially onPause?

I tried this, but not working
@Override
    public void onPause() {
        super.onPause();
        stopLocationUpdates();
    }

Never mind. Found stopLocationUpdates().


bug(MatTable MatSort MatPaginator): Wrong display of sticky columns after sorting

Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
No response
Description
Example project: https://stackblitz.com/edit/angular-ivy-fojmnq?file=src/app/app.component.ts
A MatTable with sticky columns, MatSort and MatPaginator are created. After scrolling the table horizontally and clicking on sort header, some cells of sticky columns are rendered with offsets. After moving a cursor or tapping (on touch devices) the view is refreshed and looks as expected.
This issue appears only when not all of data present in MatTableSource.data is displayed. When page size of paginator is set that is show all the data, everything works fine.
Reproduction
Steps to reproduce in example project:

Move horizontal scroll
Click on any sort header 2 times - try not to move cursor much
Sticky columns cells should display with offset or not display at all
After moving cursor, view is back to normal

Expected Behavior
Sticky columns shouldn't change position after sorting data
Actual Behavior
Sticky columns change position
Environment

Angular: 13.2.2
CDK/Material: 13.2.2
Browser(s): Chrome (v101.0.4951.64), Firefox(v100.0.2), Safari (v15.1)
Operating System (e.g. Windows, macOS, Ubuntu): MacOS

I also noticed this issue in version 13.3.7 when filtering a table with sticky columns.
This behavior used to work in a previous version of CDK: 13.1.1 and this issue seems to have been introduced in 13.1.2 this commit.

if you want workaround for now I guess you can do something like this at the end of the request
i know it's not the prettiest thing in the world but it works until they fix it
      setTimeout(() => {
        document.body.click();
      }, 0);


update axios

Updates axios to the latest version to resolve a vulnerability and bring in the newest fixes/features.
This also updates our documentation and typing around cookie support, as it is currently inaccurate.

:tada: This PR is included in version 2.17.11 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[TypeReconstruction] Don't choke on slightly invalid mangled names.

This was taken from a real project, but I don't have it anymore
and I can't reproduce it on newer versions of swift (probably
something changed in the DWARF type mangling). In any case, as
this can be triggered by user input, we shouldn't assert, and
error out instead.
rdar://problem/39283717

@swift-ci please test and merge

FWIW, this gets demangled something weird, I don't think it's really valid, but cc:ing @eeckstein for confirmation
$S12EyeCandyCore11XPCListenerC14messageHandleryyAA13XPCConnectionV_AA10XPCMessageVxtcvpfiyAF_AHxtcfU_TA.4 ---> closure #1 (EyeCandyCore.XPCConnection, EyeCandyCore.XPCMessage, A) -> () in variable initialization expression of EyeCandyCore.XPCListener.messageHandler : (EyeCandyCore.XPCConnection, EyeCandyCore.XPCMessage, A) -> ()partial apply forwarder with unmangled suffix ".4"


MetaQA relation match

thanks,
I wonder to know how to use relation match in KGQA/LSTM, when use 3 hops

and where is "Pruning_traing/test" for MetaQA?


feat: add internalWasmUrl option

users can provide their own wasm path to meet the needs of some special situations.

I ues it  with  etag, to void cache.


fix(org): #+ constructs hidden forever

Leaving org-tree-slide-mode did not unhide #+ constructs.
Now +org-present-hide-blocks-h is triggered when leaving the mode as
well, fixing the behaviour.

Thanks for your help!


[Release-Patch-Part2] Prepare Patch Release for February 2024

Include,

Storage
Identity-Extension
SchemaRegistry
Container Registry
ACS PhoneNumber, JobRouter

API change check
API changes are not detected in this pull request.


One Typo

Should be VMware PowerCLI

@v-DavidTang : Thanks for your contribution to the Azure documentation! The author, @rayne-wiselman, has been notified to review your proposed change.


Update to newest Alpine and Java

Bumping this one :)

I bumped the Java version again.
Could someone merge this please?
There's been a lot of security fixes in Alpine and Java,
so it would be good to have a more secure base image with those fixes.

+1 @nhuray would appreciate that too


Adjust styling table formatting and bright/non-bright terminal colours

I've created a PR https://github.com/tinted-theming/home/pull/110 which makes the bright/non-bright situation a bit more explicit for Base16.
Since Base24 is a superset of base16, we should theoretically be able to use a base24 colour palette for base16 and it would generate the same thing. The way things are now Base24's base08 - base0E is considered "not bright", however the colours used here tend to be less saturated than base10 - base17 - so in practise Base24's "bright" terminal colours are typically less bright.
People also tend make themes with highly saturated colours, so adding a "bright" variant becomes more difficult than adding a less saturated variant. Of course this isn't always the case, there can be themes that have low saturation, and in those cases a bright variant would be easy to add and a less saturated variant would be more difficult to add, so things won't always work out perfectly.
With this in mind, what are your thoughts with this edit? Do you agree with the above, etc?

Yeah I think I'm happy with this. Makes a lot of sense given the use of saturated colours for base 16.
This is a backwards incompatible change imo with the old base24. That is the base24 schemes/ theme templates will need swapping around to accommodate this. Particularly for terminal themes.
I've not a problem with that, just something we need to keep in mind. I'm wondering how best we can communicate this. To me, it's a major version change and possibly a notice on the repo highlighting this

Hmm, I guess it comes down to whether we want bright colours to be used for “Variables, XML Tags, Markup Link Text, Markup Lists, Diff Deleted” etc or if those should be less saturated. Maybe it makes sense to have those as less saturated and we just change the colours for the base24 variant and be ok that the base24 scheme would look different if used for building a base16 template than building the base16 template using the corresponding base16 scheme.

If we're using the automated builders, as long as we merge template changes and scheme updates before the automated builders run, we can at least make it work for all our "official" templates.

Yeah I'm absolutely happy with that then.

What do you two think about bright colours being base08-base0E vs the “non bright” colours? I’m asking because I don’t actually have a strong opinion on it, just as long as it’s hinted at which one is the default in the spec so we can carry that across to base24. So I can merge the PR, but  I just want to hear what you think about that decision in particular, even if your feeling is “doesn’t matter to me”.

Closing this PR since I forgot about this PR and ended up creating another PR: https://github.com/tinted-theming/base24/pull/30 😅


add notebook container

Do we still need this PR? As we are using bitnami/spark now?

yeah, I think this repo is redundant. You should check with what Boris is doing - he has a jupyter container running in the cdm stack, not sure if it's working. If not, you might want to consider building a notebook image on top of the bitnami image, but I'd make a new repo for that


Object.require is deprecated.

Use remote.BrowserWindow instead of remote.require("browser-window")
Object.require (/Applications/Atom.app/Contents/Resources/app.asar/src/electron-shims.js:54:12)
Object.<anonymous> (/Users/jasonloomis/.atom/packages/integrated-learn-environment/lib/models/browser-window-wrapper.coffee:9:26)
Object.<anonymous> (/Users/jasonloomis/.atom/packages/integrated-learn-environment/lib/models/browser-window-wrapper.coffee:71:4)
Module._compile (/Applications/Atom.app/Contents/Resources/app.asar/src/native-compile-cache.js:109:30)
Object..coffee (/Applications/Atom.app/Contents/Resources/app.asar/src/compile-cache.js:216:21)
Module.load (module.js:473:32)

This deprecation warning is coming from the integrated-learn-environment package which is actually no longer maintained. You can go ahead and uninstall it.


Swift 3 support

This pull request was created to give the Swift 3 support to this library

Thanks so much @sampaioroberto for the PR! 🙌 Really means a lot! 🙇 Unfortunately I can't really get to this today, but will try to check this ASAP.

Ok, man! Take ur time 😃

I took my time, huh! Sorry for the delay, I'm gonna merge it now and work on Swift 3.0 update! Thank you so much once again! 🎉🎉🎉


Directory issue in running node server.

During the setup of utility emmission channel setup in my ubuntu 18.04 i faced an issue at point. seeding the fabric database.I am trying to run the node server by using command node egrid-data-loader.js load_utility_emissions eGRID2018_Data_v2.xlsx NRL18 but i am getting the issue.

It is asking for a peerOrganisation folder inside organisation folder which is not there.
Also I observer Organisation/peerOrganisation/* is on gitignore.
How to solve this issue? How to get the peer organisation folder? How to run the node script to get the data ?

Organisation/peerOrganisation/* are flooded with crypto related folder (prvKey , signcert) for each organisation after successful execution of 'Generating crpyto and orgs...' script here , from the log I can say start.sh failed to generate crypto files .

Could you verify all docker container are running
Please Share initial log after running script.sh.

Organisation/peerOrganisation/* are flooded with crypto related folder (prvKey , signcert) for each organisation after successful execution of 'Generating crpyto and orgs...' script here , from the log I can say start.sh failed to generate crypto files .

Could you verify all docker container are running
Please Share initial log after running script.sh.

I think you are right. Because I was bit confused to the directory where to run sh start.sh. Now when I run this command at "/docker-compose-setup" I get this log..
`bhaarat@khatri-inspiron-3585:~/Desktop/lfx/blockchain-carbon-accounting/utility-emissions-channel/docker-compose-setup$ sh start.sh
Starting blockchain-carbon-accounting repo
Checking for carbonAccounting network...
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks: dial unix /var/run/docker.sock: connect: permission denied
"docker network inspect" requires at least 1 argument.
See 'docker network inspect --help'.
Usage:  docker network inspect [OPTIONS] NETWORK [NETWORK...]
Display detailed information on one or more networks
Starting docker network with name carbonAccounting
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/swarm/init: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/create: dial unix /var/run/docker.sock: connect: permission denied
Starting CAs...
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
PermissionError: [Errno 13] Permission denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "requests/adapters.py", line 449, in send
File "urllib3/connectionpool.py", line 727, in urlopen
File "urllib3/util/retry.py", line 410, in increment
File "urllib3/packages/six.py", line 734, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker/api/client.py", line 214, in _retrieve_server_version
File "docker/api/daemon.py", line 181, in version
File "docker/utils/decorators.py", line 46, in inner
File "docker/api/client.py", line 237, in _get
File "requests/sessions.py", line 543, in get
File "requests/sessions.py", line 530, in request
File "requests/sessions.py", line 643, in send
File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 3, in
File "compose/cli/main.py", line 81, in main
File "compose/cli/main.py", line 200, in perform_command
File "compose/cli/command.py", line 70, in project_from_options
File "compose/cli/command.py", line 153, in get_project
File "compose/cli/docker_client.py", line 43, in get_client
File "compose/cli/docker_client.py", line 170, in docker_client
File "docker/api/client.py", line 197, in init
File "docker/api/client.py", line 222, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[17867] Failed to execute script docker-compose
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Starting client...
Using volumes for Cli container:
/home/bhaarat/Desktop/lfx/blockchain-carbon-accounting/utility-emissions-channel/docker-compose-setup
/home/bhaarat/Desktop/lfx/blockchain-carbon-accounting/utility-emissions-channel/chaincode
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create?name=cli: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.
Generating crpyto and orgs...
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/cli/json: dial unix /var/run/docker.sock: connect: permission denied
Starting orderers, peers, and couchdb...
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
PermissionError: [Errno 13] Permission denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "requests/adapters.py", line 449, in send
File "urllib3/connectionpool.py", line 727, in urlopen
File "urllib3/util/retry.py", line 410, in increment
File "urllib3/packages/six.py", line 734, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker/api/client.py", line 214, in _retrieve_server_version
File "docker/api/daemon.py", line 181, in version
File "docker/utils/decorators.py", line 46, in inner
File "docker/api/client.py", line 237, in _get
File "requests/sessions.py", line 543, in get
File "requests/sessions.py", line 530, in request
File "requests/sessions.py", line 643, in send
File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 3, in
File "compose/cli/main.py", line 81, in main
File "compose/cli/main.py", line 200, in perform_command
File "compose/cli/command.py", line 70, in project_from_options
File "compose/cli/command.py", line 153, in get_project
File "compose/cli/docker_client.py", line 43, in get_client
File "compose/cli/docker_client.py", line 170, in docker_client
File "docker/api/client.py", line 197, in init
File "docker/api/client.py", line 222, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[17926] Failed to execute script docker-compose
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Creating the channel...
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/cli/json: dial unix /var/run/docker.sock: connect: permission denied
Install ext chaincode...
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/cli/json: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/cli/json: dial unix /var/run/docker.sock: connect: permission denied
Starting ext chaincode...
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
PermissionError: [Errno 13] Permission denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "requests/adapters.py", line 449, in send
File "urllib3/connectionpool.py", line 727, in urlopen
File "urllib3/util/retry.py", line 410, in increment
File "urllib3/packages/six.py", line 734, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1277, in request
File "http/client.py", line 1323, in _send_request
File "http/client.py", line 1272, in endheaders
File "http/client.py", line 1032, in _send_output
File "http/client.py", line 972, in send
File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker/api/client.py", line 214, in _retrieve_server_version
File "docker/api/daemon.py", line 181, in version
File "docker/utils/decorators.py", line 46, in inner
File "docker/api/client.py", line 237, in _get
File "requests/sessions.py", line 543, in get
File "requests/sessions.py", line 530, in request
File "requests/sessions.py", line 643, in send
File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 3, in
File "compose/cli/main.py", line 81, in main
File "compose/cli/main.py", line 200, in perform_command
File "compose/cli/command.py", line 70, in project_from_options
File "compose/cli/command.py", line 153, in get_project
File "compose/cli/docker_client.py", line 43, in get_client
File "compose/cli/docker_client.py", line 170, in docker_client
File "docker/api/client.py", line 197, in init
File "docker/api/client.py", line 222, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[18049] Failed to execute script docker-compose
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/networks/carbonAccounting/connect: dial unix /var/run/docker.sock: connect: permission denied
Deploying CC Ext...
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/cli/json: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/cli/json: dial unix /var/run/docker.sock: connect: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/cli/json: dial unix /var/run/docker.sock: connect: permission denied
Starting the api...
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/json?filters=%7B%22reference%22%3A%7B%22api%22%3Atrue%7D%7D: dial unix /var/run/docker.sock: connect: permission denied
Building api image
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=api&target=&ulimits=null&version=1: dial unix /var/run/docker.sock: connect: permission denied
Using volumes for API container:
/home/bhaarat/Desktop/lfx/blockchain-carbon-accounting/utility-emissions-channel/docker-compose-setup
/home/bhaarat/Desktop/lfx/blockchain-carbon-accounting/utility-emissions-channel/typescript_app
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create?name=api: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.`
What's the issue??? here

You don't have a proper permissions for running docker in a non privileged mode.
to fix on ubuntu
sudo groupadd docker
sudo usermod -aG docker $USER

then restart
more details : https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user

You don't have a proper permissions for running docker in a non privileged mode.
to fix on ubuntu
sudo groupadd docker
sudo usermod -aG docker $USER

then restart
more details : https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user

Thanks @Zzocker  I tried this and the permission error which i was getting previously was gone. But then after it I was getting some another error I am not sure but I think that error was because I didn't follow the point number 3 of https://github.com/hyperledger-labs/blockchain-carbon-accounting/blob/main/utility-emissions-channel/README.md#running-the-fabric-network-and-express-api.
So today when i was following that point i was bit confused because I guess there is some updation in this documentation.
I have created account on infura and then created one ethereum project but where i have to put that project id and secret now??
I have also downloaded the metamask extension and need to setup now.. I am basically confused in the third point of the documentation. Please help me out here...
Beside from it , after pulling new main branch I tried to run this command sh ./scripts/reset.sh && sh start.sh  but getting these error now.
bhaarat@khatri-inspiron-3585:~/Desktop/lfx/blockchainbhaarat@khatri-inspiron-3585:~/Desktop/lfx/blockchain-carbon-accounting/utility-emissions-channel/docker-cbhaarat@khatri-inspiron-3585:~/Desktop/lfx/blockchain-carbon-accounting/utility-emissions-channel/docker-cbhaarat@khatri-inspiron-3585:~/Desktop/lfx/blockchainbhaarat@khatri-inspiron-3585:~/Desktop/lfx/blockchain-carbon-accounting/utility-emissions-channel/docker-cbhaarat@khatri-inspiron-3585:~/Desktop/lfx/blockchain-carbon-accounting/utility-emissions-channel/docker-compose-setup$ sh ./scripts/reset.sh && sh start.sh
Stopping network

Error response from daemon: No such container: cli
WARNING: The COMPOSE_PROJECT_NAME variable is not set. Defaulting to a blank string.
Removing network docker_carbonAccounting
WARNING: Network docker_carbonAccounting not found.
Network net_carbonAccounting is external, skipping
Removing volume docker_orderer1.auditor1.carbonAccounting.com
WARNING: Volume docker_orderer1.auditor1.carbonAccounting.com not found.
Removing volume docker_orderer1.auditor2.carbonAccounting.com
WARNING: Volume docker_orderer1.auditor2.carbonAccounting.com not found.
Removing volume docker_orderer1.auditor3.carbonAccounting.com
WARNING: Volume docker_orderer1.auditor3.carbonAccounting.com not found.
Removing volume docker_peer1.auditor1.carbonAccounting.com
WARNING: Volume docker_peer1.auditor1.carbonAccounting.com not found.
Removing volume docker_peer1.auditor2.carbonAccounting.com
WARNING: Volume docker_peer1.auditor2.carbonAccounting.com not found.
Removing volume docker_peer1.auditor3.carbonAccounting.com
WARNING: Volume docker_peer1.auditor3.carbonAccounting.com not found.
Removing volume docker_pgdata
WARNING: Volume docker_pgdata not found.
Removing volume docker_walletstore
WARNING: Volume docker_walletstore not found.
---- No containers available for deletion ----
---- No images available for deletion ----
rm: cannot remove 'emissionscontract.tar.gz': No such file or directory
Resetting wallets and fabric-ca-server dbs...
Wallets at ../typescript_app/dist/src/blockchain-gateway/wallets do not exist, skipping.
db for auditor1 does not exist, skipping.
db for auditor2 does not exist, skipping.
db for auditor3 does not exist, skipping.
Finished removing all fabric-ca-server dbs.
Pruning volumes...
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] Total reclaimed space: 0B
Removing build under typescript_app/dist...
rm: cannot remove 'chaincode*_log.txt': No such file or directory

I am still trying to understand this error.
Any help would be appreciable.
Thanks :)

You don't need infura project for running this on your local machine.
cd docker-compose-setup
./bootstrap.sh  2.2.1 1.4.9 -d -s /// make sure you run this
./scripts/fabricNetwork.sh reset

Start local Hardhat network
cd blockchain-carbon-accounting/net-emissions-token-network/
npx hardhat node

Finally
cd docker-compose-setup && ./start

Please let me know if you face any error .

You don't need infura project for running this on your local machine.
cd docker-compose-setup
./bootstrap.sh  2.2.1 1.4.9 -d -s /// make sure you run this
./scripts/fabricNetwork.sh reset

Start local Hardhat network
cd blockchain-carbon-accounting/net-emissions-token-network/
npx hardhat node

Finally
cd docker-compose-setup && ./start

Please let me know if you face any error .

Thanks @Zzocker for your help server started working now.
Sorry for the late response I am busy with my 4th sem exam a bit.
Now CouchDB is running perfectly.I have loaded the files using node commands mentioned and make some queries to couchDB.
I have implemented majority thing mention at here .
Can You suggest me some documentry so that I can understand what is happening actually inside Utility Emmission Directory
OR suggest me some more good first issue in which I can work.
Thank You

Please close this issue if your issue has be resolved.
To get more understating about the project ,

Start with Si video  : https://www.youtube.com/watch?v=zIYfjF4U2G8&feature=youtu.be
Try running and understanding end-to-end test : https://github.com/hyperledger-labs/blockchain-carbon-accounting/blob/main/utility-emissions-channel/typescript_app/tests/e2e.test.ts

Thank you

Ok sure.
Thanks :-)


Add Faculty class

Add Faculty and its test class.

Now I store the faculty value as an enum. Do I need to change it to String?


stop stream when close publisher

!!! Before submitting a new feature request, please ensure you have searched for any existing features and utilized
the Ask AI feature at https://ossrs.io or https://ossrs.net (for users in China). Duplicate issues or
questions that are overly simple or already addressed in the documentation will be removed without any
response.
When I called srs.sdk.js to close the pc, I found that the browser page was still using the camera
What is the business background? Please provide a description.
Who are the users? How do they utilize this feature? What problem does this feature address?
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
// Close the publisher.  stop stream when close publisher
self.close = function () {
self.pc && self.pc.close()
self.pc = null
// If a local stream exists, stop all of its tracks.
if (self.stream) {
console.log('close publisher stream')
self.stream.getTracks().forEach(track => track.stop())
self.stream = null
}
}
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
TRANS_BY_GPT4

I recently switched to SRT and the SRS_VHOST_PUBLISH_KICKOFF_FOR_IDLE=5 stopped working. It seems that SRT is not detecting that there are 0 people watching...

It seems that the first connection is preventing the stream from stopping. There is always 1 client connected to the stream. Even before the on_play event got triggered.


fix: test

ruff and golangci-lint not working right now, others should work

thanks! bunch of path bs. How did you figure it out?

thanks! bunch of path bs. How did you figure it out?

A bunch of which and echos lol, and maybe some patience.
act was super nice tho, probably halfen the debug time and the terminal is just so much more asthetic than the webui

Ok awesome. any insight into why ruff and golangci-lint don't work despite being able to debug with act?

Oh ktlint also doesn't work anymore. The linters returned an empty table so I'm guessing either the test or the config is not correct. Formatters kinda the same story. Not an installation issue, they passed the executable assertion. I will try to tweak the tests tomorrow to see if that helps.

Ok, make sure to watch out for things getting sent to stdout vs stderr

ruff --fix has the same issue as prettierd, others are unclear


Fix Error: write after end

When a HEAD request is sent to the ReactQL server, it's easy to get:

Error: write after end
at writeAfterEnd (_stream_writable.js:236:12)
at PassThrough.Writable.write (_stream_writable.js:287:5)
at ReactMarkupReadableStream.ondata (_stream_readable.js:640:20)
at emitOne (events.js:115:13)
at ReactMarkupReadableStream.emit (events.js:210:7)
at ReactMarkupReadableStream.Readable.read (_stream_readable.js:476:10)
at flow (_stream_readable.js:847:34)
at PassThrough. (_stream_readable.js:708:7)
at emitNone (events.js:105:13)
at PassThrough.emit (events.js:207:7)

Likely because the client has disconnected early, and we're trying to pipe output to a non-existent socket.
Output should only be sent if the client is still around.

Until this is added to the next version, this can be fixed with the following kit/entry/server.js file:
/* eslint-disable no-param-reassign, no-console */

// Server entry point, for Webpack.  This will spawn a Koa web server
// and listen for HTTP requests.  Clients will get a return render of React
// or the file they have requested.

// ----------------------
// IMPORTS

/* Node */

// For pre-pending a `<!DOCTYPE html>` stream to the server response
import stringToStream from 'string-to-stream';
import multiStream from 'multistream';

// HTTP & SSL servers.  We can use `config.enableSSL|disableHTTP()` to enable
// HTTPS and disable plain HTTP respectively, so we'll use Node's core libs
// for building both server types.
import http from 'http';
import https from 'https';

/* NPM */

// Patch global.`fetch` so that Apollo calls to GraphQL work
import 'isomorphic-fetch';

// React UI
import React from 'react';

// React utility to transform JSX to HTML (to send back to the client)
import ReactDOMServer from 'react-dom/server';

// Koa 2 web server.  Handles incoming HTTP requests, and will serve back
// the React render, or any of the static assets being compiled
import Koa from 'koa';

// Apollo tools to connect to a GraphQL server.  We'll grab the
// `ApolloProvider` HOC component, which will inject any 'listening' React
// components with GraphQL data props.  We'll also use `getDataFromTree`
// to await data being ready before rendering back HTML to the client
import { ApolloProvider, getDataFromTree } from 'react-apollo';

// Enforce SSL, if required
import koaSSL from 'koa-sslify';

// Enable cross-origin requests
import koaCors from 'kcors';

// Static file handler
import koaSend from 'koa-send';

// HTTP header hardening
import koaHelmet from 'koa-helmet';

// Koa Router, for handling URL requests
import KoaRouter from 'koa-router';

// High-precision timing, so we can debug response time to serve a request
import ms from 'microseconds';

// React Router HOC for figuring out the exact React hierarchy to display
// based on the URL
import { StaticRouter } from 'react-router';

// <Helmet> component for retrieving <head> section, so we can set page
// title, meta info, etc along with the initial HTML
import Helmet from 'react-helmet';

// Import the Apollo GraphQL server, for Koa
import { graphqlKoa, graphiqlKoa } from 'apollo-server-koa';

// Allow local GraphQL schema querying when using a built-in GraphQL server
import apolloLocalQuery from 'apollo-local-query';

// Import all of the GraphQL lib, for use with our Apollo client connection
import * as graphql from 'graphql';

/* ReactQL */

// App entry point.  This must come first, because app.js will set-up the
// server config that we'll use later
import App from 'src/app';

// Custom redux store creator.  This will allow us to create a store 'outside'
// of Apollo, so we can apply our own reducers and make use of the Redux dev
// tools in the browser
import createNewStore from 'kit/lib/redux';

// Initial view to send back HTML render
import Html from 'kit/views/ssr';

// Grab the shared Apollo Client / network interface instantiation
import { getNetworkInterface, createClient } from 'kit/lib/apollo';

// App settings, which we'll use to customise the server -- must be loaded
// *after* app.js has been called, so the correct settings have been set
import config from 'kit/config';

// Import paths.  We'll use this to figure out where our public folder is
// so we can serve static files
import PATHS from 'config/paths';

// ----------------------

// Create a network layer based on settings.  This is an immediate function
// that binds either the `localInterface` function (if there's a built-in
// GraphQL) or `externalInterface` (if we're pointing outside of ReactQL)
const createNeworkInterface = (() => {
  // For a local interface, we want to allow passing in the request's
  // context object, which can then feed through to our GraphQL queries to
  // extract pertinent information and manipulate the response
  function localInterface(context) {
    return apolloLocalQuery.createLocalInterface(
      graphql,
      config.graphQLSchema,
      {
        // Attach the request's context, which certain GraphQL queries might
        // need for accessing cookies, auth headers, etc.
        context,
      },
    );
  }

  function externalInterface(ctx) {
    return getNetworkInterface(config.graphQLEndpoint, ctx.apollo.networkOptions);
  }

  return config.graphQLServer ? localInterface : externalInterface;
})();

// Static file middleware
export function staticMiddleware() {
  return async function staticMiddlewareHandler(ctx, next) {
    try {
      if (ctx.path !== '/') {
        return await koaSend(
          ctx,
          ctx.path,
          process.env.NODE_ENV === 'production' ? {
            root: PATHS.public,
            immutable: true,
          } : {
            root: PATHS.distDev,
          },
        );
      }
    } catch (e) { /* Errors will fall through */ }
    return next();
  };
}

// Function to create a React handler, per the environment's correct
// manifest files
export function createReactHandler(css = [], scripts = [], chunkManifest = {}) {
  return async function reactHandler(ctx) {
    const routeContext = {};

    // Generate the HTML from our React tree.  We're wrapping the result
    // in `react-router`'s <StaticRouter> which will pull out URL info and
    // store it in our empty `route` object
    const components = (
      <StaticRouter location={ctx.request.url} context={routeContext}>
        <ApolloProvider store={ctx.store} client={ctx.apollo.client}>
          <App />
        </ApolloProvider>
      </StaticRouter>
    );

    // Wait for GraphQL data to be available in our initial render,
    // before dumping HTML back to the client
    await getDataFromTree(components);

    // Handle redirects
    if ([301, 302].includes(routeContext.status)) {
      // 301 = permanent redirect, 302 = temporary
      ctx.status = routeContext.status;

      // Issue the new `Location:` header
      ctx.redirect(routeContext.url);

      // Return early -- no need to set a response body
      return;
    }

    // Handle 404 Not Found
    if (routeContext.status === 404) {
      // By default, just set the status code to 404.  Or, we can use
      // `config.set404Handler()` to pass in a custom handler func that takes
      // the `ctx` and store

      if (config.handler404) {
        config.handler404(ctx);

        // Return early -- no need to set a response body, because that should
        // be taken care of by the custom 404 handler
        return;
      }

      ctx.status = routeContext.status;
    }

    // Create a stream of the React render. We'll pass in the
    // Helmet component to generate the <head> tag, as well as our Redux
    // store state so that the browser can continue from the server
    const reactStream = ReactDOMServer.renderToNodeStream(
      <Html
        helmet={Helmet.renderStatic()}
        window={{
          webpackManifest: chunkManifest,
          __STATE__: ctx.store.getState(),
        }}
        css={css}
        scripts={scripts}>
        {components}
      </Html>,
    );

    // Pipe the React stream to the HTML output
    // reactStream.pipe(htmlStream);

    // Set the return type to `text/html`, and stream the response back to
    // the client
    ctx.type = 'text/html';
    ctx.body = multiStream([stringToStream('<!DOCTYPE html>'), reactStream]);
  };
}

// Build the router, based on our app's settings.  This will define which
// Koa route handlers
const router = (new KoaRouter())
  // Set-up a general purpose /ping route to check the server is alive
  .get('/ping', async ctx => {
    ctx.body = 'pong';
  })

  // Favicon.ico.  By default, we'll serve this as a 204 No Content.
  // If /favicon.ico is available as a static file, it'll try that first
  .get('/favicon.ico', async ctx => {
    ctx.status = 204;
  });

// Build the app instance, which we'll use to define middleware for Koa
// as a precursor to handling routes
const app = new Koa()
  // Adds CORS config
  .use(koaCors(config.corsOptions))

  // Error wrapper.  If an error manages to slip through the middleware
  // chain, it will be caught and logged back here
  .use(async (ctx, next) => {
    try {
      await next();
    } catch (e) {
      // If we have a custom error handler, use that - else simply log a
      // message and return one to the user
      if (typeof config.errorHandler === 'function') {
        config.errorHandler(e, ctx, next);
      } else {
        console.log('Error:', e.message);
        ctx.body = 'There was an error. Please try again later.';
      }
    }
  });

if (config.enableTiming) {
  // It's useful to see how long a request takes to respond.  Add the
  // timing to a HTTP Response header
  app.use(async (ctx, next) => {
    const start = ms.now();
    await next();
    const end = ms.parse(ms.since(start));
    const total = end.microseconds + (end.milliseconds * 1e3) + (end.seconds * 1e6);
    ctx.set('Response-Time', `${total / 1e3}ms`);
  });
}

// Middleware to set the per-request environment, including the Apollo client.
// These can be overriden/added to in userland with `config.addBeforeMiddleware()`
app.use(async (ctx, next) => {
  ctx.apollo = {};
  return next();
});

// Add 'before' middleware that needs to be invoked before the per-request
// Apollo client and Redux store has instantiated
config.beforeMiddleware.forEach(middlewareFunc => app.use(middlewareFunc));

// Create a new Apollo client and Redux store per request.  This will be
// stored on the `ctx` object, making it available for the React handler or
// any subsequent route/middleware
app.use(async (ctx, next) => {
  // Create a new server Apollo client for this request, if we don't already
  // have one
  if (!ctx.apollo.client) {
    ctx.apollo.client = createClient({
      ssrMode: true,
      // Create a network request.  If we're running an internal server, this
      // will be a function that accepts the request's context, to feed through
      // to the GraphQL schema
      networkInterface: createNeworkInterface(ctx),
      ...ctx.apollo.options,
    });
  }

  // Create a new Redux store for this request, if we don't have one
  if (!ctx.store) {
    ctx.store = createNewStore(ctx.apollo.client);
  }

  // Pass to the next middleware in the chain: React, custom middleware, etc
  return next();
});

/* FORCE SSL */

// Middleware to re-write HTTP requests to SSL, if required.
if (config.enableForceSSL) {
  app.use(koaSSL(config.enableForceSSL));
}

// Middleware to add preliminary security for HTTP headers via Koa Helmet
if (config.enableKoaHelmet) {
  app.use(koaHelmet(config.koaHelmetOptions));
}

// Attach custom middleware
config.middleware.forEach(middlewareFunc => app.use(middlewareFunc));

// Attach an internal GraphQL server, if we need one
if (config.graphQLServer) {
  // Attach the GraphQL schema to the server, and hook it up to the endpoint
  // to listen to POST requests
  router.post(
    config.graphQLEndpoint,
    graphqlKoa(context => ({
      // Bind the current request context, so it's accessible within GraphQL
      context,
      // Attach the GraphQL schema
      schema: config.graphQLSchema,
    })),
  );
}

// Do we need the GraphiQL query interface?  This can be used if we have an
// internal GraphQL server, or if we're pointing to an external server.  First,
// we check if `config.graphiql` === `true` to see if we need one...

if (config.graphiQL) {
  // The GraphiQL endpoint default depends on this order of precedence:
  // explicit -> internal GraphQL server endpoint -> /graphql
  let graphiQLEndpoint;

  if (typeof config.graphiQL === 'string') {
    // Since we've explicitly passed a string, we'll use that as the endpoint
    graphiQLEndpoint = config.graphiQL;
  } else if (config.graphQLServer) {
    // If we have an internal GraphQL server, AND we haven't set a string,
    // the default GraphiQL path should be the same as the server endpoint
    graphiQLEndpoint = config.graphQLEndpoint;
  } else {
    // Since we haven't set anything, AND we don't have an internal server,
    // by default we'll use `/graphql` which will work for an external server
    graphiQLEndpoint = '/graphql';
  }

  router.get(
    graphiQLEndpoint,
    graphiqlKoa({
      endpointURL: config.graphQLEndpoint,
    }),
  );
}

// Attach any custom routes we may have set in userland
config.routes.forEach(route => {
  router[route.method](route.route, ...route.handlers);
});

/* BODY PARSING */

// `koa-bodyparser` is used to process POST requests.  Check that it's enabled
// (default) and apply a custom config if we need one
if (config.enableBodyParser) {
  app.use(require('koa-bodyparser')(
    // Pass in any options that may have been set in userland
    config.bodyParserOptions,
  ));
}

/* CUSTOM APP INSTANTIATION */

// Pass the `app` to do anything we need with it in userland. Useful for
// custom instantiation that doesn't fit into the middleware/route functions
if (typeof config.koaAppFunc === 'function') {
  config.koaAppFunc(app);
}

// Listener function that will start http(s) server(s) based on userland
// config and available ports
const listen = () => {
  // Spawn the listeners.
  const servers = [];

  // Plain HTTP
  if (config.enableHTTP) {
    servers.push(
      http.createServer(app.callback()).listen(process.env.PORT),
    );
  }

  // SSL -- only enable this if we have an `SSL_PORT` set on the environment
  if (process.env.SSL_PORT) {
    servers.push(
      https.createServer(config.sslOptions, app.callback()).listen(process.env.SSL_PORT),
    );
  }

  return servers;
};

// Export everything we need to run the server (in dev or prod)
export default {
  router,
  app,
  listen,
};

Apologies for the generic response, but I've released a new v3.0 kit, which has been re-written from the ground up.
It bumps the ReactQL stack to Webpack 4, React 16.4, Apollo Client 2.0, and adds Styled Components, full SSR (even in development), and has been re-written in Typescript. It's leaner, faster and has been brought completely up-to-date.
If you're still using ReactQL (and I appreciate you may not be, given the time lag in my response), I'd recommend checking out the new kit to see if your issue has been answered. If not, please feel free to post a new issue at https://github.com/leebenson/reactql


Media Picker v3 support

Media picker v3 stores the available media types so presumably SuperValueConverters could be extended to support that too?

Yes absolutely!
The media picker converter previously had to be removed in V8 due to some breaking changes - see discussion here https://github.com/umbraco/Umbraco-CMS/pull/6034.
Will happily accept a PR 😊


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v4.14 to before v5.15.13",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=74310e06be4d74dcf67cd108366710dee5c576d5",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=17691bada6b2f1d5f1c0f6d28cd9d0727023b0ff"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "74310e06be4d74dcf67cd108366710dee5c576d5",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "17691bada6b2f1d5f1c0f6d28cd9d0727023b0ff",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "binder: fix async_free_space accounting for empty parcels\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.15.13 by commit 17691bada6b2f1d5f1c0f6d28cd9d0727023b0ff, it was introduced in version v4.14 by commit 74310e06be4d74dcf67cd108366710dee5c576d5. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2022-1000056


feat(Topology): continuity from a product with a discrete space

Add four pairs of lemmas continuous((Within)At/On)_prod_of_discrete_left/right  in ContinuousOn.lean: to check continuity of a function from X × Y to Z with X discrete, it suffices to check continuity of every slice of it with x : X fixed.

Remove duplicate lemmas continuous_uncurry_of_discreteTopology(_left) from Constructions.lean in favor of the more general (iff) version.

Move the lemma continuous_iff_continuousOn_univ up.

Thanks!
bors merge


Streaming logs

https://app.asana.com/0/58153350582042/58211982755442/f
Closes https://github.com/remind101/conveyor/issues/15
This adds streaming logs backed by CloudWatch Logs.
TODO

[x] Use some binary sequence when closing the log stream, so that readers can close.
[x] Set log retention to 7 days.
[x] Lock down ACL to only allow access to the provisioned log group.

:+1:


readthedocs CI has stopped

readthedocs isn't building PRs, despite the configuration being set in advanced settings
Maybe @choldgraf or @willingc has some insight on why this is or how to fix it?
I think we can remove the docs builds from circleci once we figure that out.

I betcha this is carryover from when we moved from jupyter/ to jupyterhub/. I just made that switch on RTD, let's see if it starts building now.

This looks like it's building on RTD @minrk. Where are you seeing it not building?

Where are you seeing it not building?

The CI status badge wasn't being added to PRs, though latest was still getting built. I see the badge now now for latest PR pushes, so I think you were right, @choldgraf. Thanks both!
I guess we need CI for our CI so we know when our builds aren't building...


Error offset iPhone 5C

sysname: Darwin
nodename: iPhone
release: 16.3.0
version: Darwin Kernel Version 16.3.0: Tue Nov 29 21:40:06 PST 2016; root:xnu-3789.32.1~4/RELEASE_ARM_S5L8950X
machine: iPhone5,4
missing offset, prob crashing

<- iphone 5c ios 10.2 , does anyone has headers of it?

This is a 32-bit device. Get out


buildSFX() no longer works...

All of the sudden, buildSFX() just up and stopped working correctly for me.
It is now including a file at the top of my bundle that does not have System.import() in it. It looks like the contents of builder/templates/sfx-core.js.
jspm --version 0.15.6
"systemjs-builder": "^0.10.6"
<EMAIL_ADDRESS><EMAIL_ADDRESS>"postinstall": "jspm install && jspm dl-loader",

@lookfirst sfx bundles don't support System.import as that implies dynamic loading, which then just becomes a normal bundle. Just let me know if you have further questions at all.

I want dynamic loading of css files.

Right - so any type of dynamic loading means using the full SystemJS with a normal bundle.

It is quite unclear that there is a 'full SystemJS' and a 'partial SystemJS' that is only for SFX bundles.
From the docs: To make a bundle that is independent of the SystemJS loader entirely, we can make SFX bundles:
That said, when I encountered this problem today, I also tried to revert my code back to just a simple build() and append es6 loader and system.js and got all sorts of weird errors there too. Something is really fishy here.

Let me know if you still need help with this.


Fix some FlushMode/BatchManager related issues

Removing the BatchManager added a regression as message emission was issued synchronously. Fixing this by yielding a turn before calling emit.
Remove/refactor submitCore from LocalDocumentDeltaConnection

There is no need for this change, the failure I was observing had a different root cause.


Replace reference to deprecated role

This fixes the AppScalefile that the fast-start script generates.

https://ocd.appscale.com:8080/job/Daily Build/4639/


fix: sw-card overflow and overlap elements

1. Why is this change necessary?
The filter drop shadow in administration creates a stacking-context and can conflict with some situations when an element in sw-card is overflowing and follow by another sw-card
https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Positioning/Understanding_z_index/The_stacking_context
2. What does this change do, exactly?
Change the mixins drop-shadow-default() drop shadow function from filter to box-shadow
3. Describe each step to reproduce the issue or behaviour.
I have this problem on Shopware extensions that have a sw-card after another that have a media field

4. Please link to the relevant issues (if any).
5. Checklist

[x] I have rebased my changes to remove merge conflicts
[ ] I have written tests and verified that they fail without my change
[x] I have created a changelog file with all necessary information about my changes
[ ] I have written or adjusted the documentation according to my changes
[ ] This change has comments for package types, values, functions, and non-obvious lines of code
[x] I have read the contribution requirements and fulfil them.

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/NEXT-23792
Please use this issue to track the state of your pull request.

This issue is getting awareness on StackOverflow https://stackoverflow.com/questions/74233423/z-index-problem-with-media-component-in-plugin-config/74233541

Hi @stephan4p,
thank you for your contribution. We have decided to adopt your merge request in a slightly different way. You can find the commit here. This should also fix the behavior you described.


Update issue templates

Closes #45

Long story short: overly-burdensome templates just get Ctrl-A, Backspace'd.

This has exactly been my experience in the past, as it's impossible to make a one size fits all issue or PR message.  It was only useful to remind developers (in PRs in particular) to be sure tests and docs are updated.


Rename Staging account to Preprod

To limit the number of organisation child accounts needed, rename staging to preprod - this will provide a place for other environments, like dev and test, to exist simultaneously in separate VPCs, without needing to create new accounts for each.
This is primarily to limit the cost of "expensive" resources (such as AWS PrivateLink Interface Endpoints) which can be deployed once per account and shared across these resources.

Didn't fix all cases of Staging in code, will need further changes.


-verbose false doesn't work

All of the following produce verbose output:
-verbose false
-verbose no
-verbose 0

Using CFLint-1.4.1-all.jar

That switch does not take an argument. Its existence is the only thing checked. See the wiki page for which switches take an argument.

Thanks. The readme says:

Verbose mode (-verbose ) enables verbose linting output.

Maybe we should delete the "" part?

fixed in dev branch


Add F32 newtype

Adds a newtype for f32 which can be used as an alternative to F32Ext.
The longer-term goal is to implement all of the current floating point operations provided by this library on the newtype, and use the newtype to impl F32Ext.
This will also allow us to implement traits from the num-traits crate (#57).

Codecov Report

Merging #72 (138eab0) into main (2460c5e) will decrease coverage by 6.30%.
The diff coverage is 20.71%.

@@            Coverage Diff             @@
##             main      #72      +/-   ##
==========================================
- Coverage   60.67%   54.37%   -6.31%
==========================================
  Files          35       36       +1
  Lines         707      789      +82
==========================================
  Hits          429      429
- Misses        278      360      +82

Impacted Files
Coverage Δ

src/f32ext.rs
0.00% <0.00%> (ø)

src/float.rs
0.00% <0.00%> (ø)

src/float/utils.rs
68.96% <ø> (ø)

src/quaternion.rs
0.00% <ø> (ø)

src/vector.rs
0.00% <ø> (ø)

src/float/abs.rs
100.00% <100.00%> (ø)

src/float/acos.rs
100.00% <100.00%> (ø)

src/float/asin.rs
100.00% <100.00%> (ø)

src/float/atan.rs
100.00% <100.00%> (ø)

src/float/atan2.rs
100.00% <100.00%> (ø)

... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2460c5e...138eab0. Read the comment docs.


Reproduced 79.1 for edgenext_small, a random error?

I retrain the edgenext_small using the official code and coomand with 8*V100, but a get 79.1 that has a minor gap with the paper's result 79.4. Is this gap a random error or something maybe left out? Thanks for the great work

I retrain the edgenext_small using the official code and coomand with 8*V100, but a get 79.1 that has a minor gap with the paper's result 79.4. Is this gap a random error or something maybe left out? Thanks for the great work

Hi @zhangzjn,
Thank you for using EdgeNeXt. In my opinion its normal as ImageNet-1K training from scratch do have similar standard deviation. Further, please verify if you are using the same hyperparameter setting as mentioned under implementation details in our paper. To be clear on the point, we trained our models on 8*A100 GPUs using PyTorch automatic mixed precision (AMP). Please let me know if you have any questions.


Is there an auto refresh for widget?

Can anyone help or guide me on how to get auto refresh working for widgets? I want the widgets to refresh at a certain interval (let's say every minute).

Hi, see code of widget clock.
https://github.com/angular-dashboard-framework/adf-widget-clock/blob/master/src/clock.js#L57
In your case you can put a field in your edit widget to get the timer in seconds and then in the widget controller define a interval to run the refresh at that interval.
NS

I should have been a little bit more clearer on my use case. I want to perform an ajax call every minute which occurs in my service. There is a resolver in the dashboardProvider but I don't understand how to trigger it. I pulled this from the adf-widget-weather example.
        resolve: {
          data: function(weatcherService, config){
            if (config.location){
              return weatcherService.get(config.location);
            }
          }
        }
```

You can't trigger a resolver from a widget. A resolver is loaded, when the widget is loaded. You have to call your service in a interval function e.g.:
dashboardProvider
  .widget('my-widget', {
    title: 'My Widget',
    description: 'Displays my widget',
    templateUrl: '{widgetsPath}/my-widget/src/view.html',
    resolve: {
      data: function(myService){
        return myService.load();
      }
    },
    controller: function($scope, $interval, myService, data){
      // set the data from the resolve method
      $scope.data = data;

      // method to refresh your data
      function refresh(){
        $scope.data = myService.load();
      }

      // call refresh method every minute
      var promise = $interval(refresh, 60000);

      // cancel interval on scope destroy
      $scope.$on('$destroy', function(){
        $interval.cancel(promise);
      });
    }
  });


Unitialized variable in f_read()

In f_read() function is variable declaration line:
UINT rcnt, cc, csect;
...
// few lines of code
...
for ( ; btr; /* Repeat until all data read */
rbuff += rcnt, fp->fptr += rcnt, *br += rcnt, btr -= rcnt) {
problem is that variable rcnt is not initialized and is assigned to random value.
Correct version:
UINT rcnt = 0, cc, csect;

It is initialized to 0 by the compiler.

Could you please send me information which C compiler is doing it. Because I personally doesn't know any compiler which initialize automatic(local) variables automatically to zero, because C specification is saying: The value of an uninitialized local variable in C is indeterminate and reading it can invoke undefined behavior. Please fix the bug.

Well I don't remember exactly why and how. Maybe related to the fact it is a typedef storage calss but not the subject here.
I'm agreed with you that it is better to init local variable anyway, this part of code come from original FatFs (based on release R0.12C from http://elm-chan.org/fsw/ff/arc/ff12c.zip), this library only package it to Arduino library format.
The same "issue" is present on the original code and is not an issue.
I don't know what lead you to open this specific issue anyway I will not modify this part of cade it should be raised to original source code. And it seems not be an issue as I guess if it is a problem this would be solved since a long time.

any update on this ? Else I will close it.


@pshobha97 Can you please provide more information like the version of angular you are using ?

@pshobha97 Can you please provide more information like the version of angular you are using ?
I am using angular version 4. when i do install the default.theme.css is not present in node_modules and the custom template is not reflecting. Please help.
Originally posted by @CuppaLabs in https://github.com/CuppaLabs/angular2-multiselect-dropdown/issues/561#issuecomment-808415356

@pshobha97 Please use the latest version v5.0.1 . it supports angular 11


Build docker image 0307

Signed-off-by: cheyang<EMAIL_ADDRESS>
Ⅰ. Describe what this PR does
Ⅱ. Does this pull request fix one issue?

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Codecov Report

Merging #618 (bddd1e3) into master (8c91abd) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #618   +/-   ##
=======================================
  Coverage   18.04%   18.04%
=======================================
  Files          64       64
  Lines        2727     2727
=======================================
  Hits          492      492
  Misses       2185     2185
  Partials       50       50

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c91abd...bddd1e3. Read the comment docs.


Add security overview doc

There are essentially 4 components that contribute to host security.

The initial bootstrap
Updates via the dashboard as desired
Firewall hole-punching as nodes go up/down
A custom overlay network

Initial Bootstrap
You can study the ubuntu bootstrap script that we're currently using: https://github.com/nanobox-io/bootstrap/blob/master/ubuntu.sh. High-level, in addition to installing and configuring Docker and Nanobox's agent, this will configure a brutal, default-deny firewall via iptables. It also ensure that the core software is up-to-date
Part of the bootstrap will also install a custom, virtual network driver: https://github.com/nanopack/red. This essentially provides a fully encapsulated layer 2 network over a peer-to-peer cluster of all of the host nodes of an app.
Once a host is bootstrapped, essentially you have a machine that is fully locked-down, running only Docker, the Nanobox agent, and redd (the virtual network). At this point, not even the other machines within the same network can talk to this machine.
At this point, the only thing that can actually communicate with the machine is nanobox.io. Nanobox injected a set of security credentials during the bootstrap that it will use to communicate with the nanobox agent. So now, nanobox can send commands to the agent (over a secure socket layer, using credentials that only it knows about).
Once another node comes online (after it is bootstrapped as well), nanobox will send a series of commands to each of the machines' agent, informing them to punch an explicit hole in the firewall for them to communicate. Once they're able to communicate, they will add each other to the virtual network, which will allow containers on each host to communicate. That process will continue each time a new host comes online or gets removed
Lastly, from the dashboard, you can run an update process will connect to each of the hosts' agents and ensure the agent and installed software is up-to-date


feat(nanocl:cargo): wait command

I don't think that the correct way to solve the problem see kubectl wait
We should have a more generic command that allow us to wait for Ressource, Vm or Cargo.


COCO and LVIS dataset preprocessing

Hi Authors! Thanks so much for your great work : )
I have a question on how do you combine the COCO and LVIS dataset, do you replace the same mask in COCO with LVIS mask? Thanks a lot!
Best,
Xueyan

Hi Xueyan, please refer to Sec. 4.2 of the RITM paper.


Mentor Card

[ ] Favorite
[ ] Country Flag
[x] Tags
[x] Title
[x] Description
[x] Image
[x] Channels


Update the copyright year in documentation

Proposed changes:

Fix the copyright year in documentation

Status (please check what you already did):

[x] made PR ready for code review
[ ] added some tests for the functionality
[x] updated the documentation
[x] updated the changelog

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Surinder Kumar seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Good point :D Do you mind signing our Contributors license agreement?

I am closing this. I had some user email related issue so I deleted the forked repo :(
I will send another pull request.


gulp.watch - Error: watch null EMFILE

gulp.watch is giving an error in the simplest use-case that I've set up. I've been using gulp.watch in a few projects and this error is "all-of-a-sudden". Perhaps I'm overlooking something? Are my glasses on my face but I can't find them?
Here is my gulpfile.js contents:
var gulp = require('gulp');
var less = require('gulp-less');

gulp.task('default', function() {
  gulp.watch('styles.less', ['styles']);
});

gulp.task('styles', function() {
  return gulp.src('styles.less')
  .pipe(less())
  .pipe(gulp.dest(''));
});

As you can see, this is a simple scenario where watching a less file change will result in compiling that file to css. Here's what I see in the console when I run 'gulp':
[19:19:30] Using gulpfile ~/Projects/Sandbox/fred/gulpfile.js
[19:19:30] Starting 'default'...
2016-01-23 19:19 gulp[41595] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
[19:19:30] Finished 'default' after 7.91 ms
events.js:141
  throw er; // Unhandled 'error' event
  ^

Error: watch null EMFILE
  at exports._errnoException (util.js:856:11)
  at FSEvent.FSWatcher._handle.onchange (fs.js:1296:21)

I also tried replacing the contents of the 'styles' task to just contain a console.log statement. Same results. Any thoughts on this?

I forgot to add the versions of everything I'm using:

npm: 3.5.3
node: 5.4.1
gulp:

CLI version: 3.9.0
Local version: 3.9.0

gulp-less: 3.0.5

That error is a node error for too many files open.  Guessing something is wrong with your OS.  I suggest googling around for EMFILE

I only have Sublime, the terminal, and Chrome open at the same time (nothing novel here). What I did notice is that if I close sublime and then run gulp, the process runs with no error.  And to add to the weirdness, this error is not consistent. My life.

  Good luck tracking down the problem.

@phated I appreciate the insight. Googling emfile now. Hope this helps someone else who may land here via google or something.

I have the same problem with you . finally i found that the actually problem is that if you open you project directory with command sublime . under terminal,and then run the gulp, that the gulp will cause the error.
so, you can open sublime application first, and then load you project directory, and finally start gulp in the terminal. everything will be ok！
hope the will help you

I have got the same problem now, if I close the Sublime, it works well, could somebody tell me why? is   it a problem of Sublime?

@lichenbuliren It's working now!!!  thanks a lot!!!

@lichenbuliren I had the same problem and your solution works! Thats great. I just can not understand what the real issue is here... Could you tell me why this happens?


EtcPal API requirements

Staging API documentation is deployed at https://etclabs.github.io/EtcPal/docs/stage/17

/AzurePipelines run

/AzurePipelines run


Do Not Disturb Exceptions

Currently the notification sound is still activated if there is any entry in "Calls, messages and conversations" section. Screenshot as attached.
Is it possible to not bypass the DND settings?
Tested on Samsung Galaxy Note 8, 9 and 20.

It is customizable in the app. See app settings: Customize Sounds -> Advanced: "Choose audio stream for Alarm sound".
By default the app using "Alarm stream" for sound that ignores DnD mode as well as other Android alarms. You can change the audio stream to "Ring and Notification stream" so as not to bypass the DnD mode.

I've selected "Ringtone and Notification stream" but it will still sound whenever I have either of the following settings under DND:

Calls, messages and conversations
Alarms and sounds >

I researched the issue.
There are different behaviors in Android 10 and 11. I was able to reproduce on Android 10.
In the next app version (v2.31) it will be be fixed for all android versions and regardless of the selected alarm stream.
I'll let you know when the app is published (in the coming days).

Thanks a lot for finding out the problem and look forward to the updated app.
And of course thousand thanks for taking over the project from the original developer!

The update published to Google Play. See app v2.32.

Some changes in the latest version (v2.33).
The choice of audio stream in the app settings will affect the behavior in DnD mode:

when "Ring and Notification stream" is selected, the app will not bypass DnD mode. This audio stream is set by default.
when "Alarm stream" is selected, the app will bypass DnD mode.


[UNDERTOW-1929] Increase even more the sockettimeout config to make s…

…ure that the sporadic failures we are seeing are not an indication the number is still too low

The goal of this PR is to use it to run several times the CI, to make sure that the failures we are seeing are not being caused by lack of a big enough so timeout.


pls have a plugin api whereby we can write our own interfacing for dedup etc, change our own hash to blake3 etc

What would you like to be added:
pls have a plugin api whereby we can write our own interfacing for dedup etc, change our own hash to blake3 etc
Why is this needed:

Data dedup is usually expensive to use... i dont agree.

it's expensive not to deploy

For some cases that have lots of duplicated data, may be we can use hard link, copy_file_range(), snapshot (planned) to avoid the duplicated data, also much faster.

actually if zfs is compatible will have built in dedup without this additional part BUT still we shld code our own dedup too.

Could you send a proposal or draft design on how to implement dedup and what's the API needed?

i'm not sure current inter workings of juicefs now but for the dedup design...
data -> [blake3 algo (or api) on file content] -> (juicefs internal hash?) -> [targeted hash backend server]
api is thus (filename = blake3 algo generated hash)

expanding on what i have suggested
file name = blake3(filecontent)

metadata stores
blake3 hash -> (counter value). if greater than 10000, or any N certain threshold (a config settable setting), store permanently.
this (counter value) can be increased if more dup are in, and decrease if deleted.

filename -> blake3 hash

=======

file content
blake3 hash -> file content

so basically data storage is basically just blake3 hash -> file content
and metadata can be use to store the dup.
based on experience. if there are more than a certain threshold of file, then no need to keep adding or deleting this counter value coz it's not worth it so the threshold can be using uint16 will do. max 65535. min can be anything. only delete if counter == 0

oops, i just finished reading juicefs architecture in more detail and realise may need a sort of raft coord to do the dedup part for metadata. i think this optional "addition" feature can be worth it. the performance sacrisfice will not be negligible

What kind of use case need dedup?
One idea:

calculate the checksum of all slices before upload, but we upload 4MB blocks before the slice (up to 64MB) is fully written, so we have to delay the upload and worse written throughput .
Check whether there is another slice with the same checksum (lookup the metadata engine). If yes, we need to compare the new data with existing slice to make sure the data is identical (there could be hash conflict). If the new data is identical to existing data, we increase the counter in meta engine
If not, we upload the slice into object storage, and remember the checksum into meta engine, then continue the write process.

If we doing this, the write throughput will lower, and there two more metadata operation for new slice.  For repeated data, we use a few downloads (could be cached) to replace the uploads.
To my knowledge, there is only a few use case that will have duplicated data, so we can't have these cost by default. The dedup feature must be optional, may be only optional for small files.

dedup definitely has to be optional setting but is great for user uploaded content where everyone is uploading the same thing.
this is offset by the application currently. it's the same thing your concern for processing. which now can be offloaded to juicefs.

using blake3 will not be any significantly slower. u can cache the checksum value in golang fastcache, which is just a quick compare before comparing with the metadata engine lookup.

in fact for big files are where the cost savings in saving redundant files to db (storage + bandwidth costs)
making this setting optional will make juicefs perfect.


"read-only filesystem" errors when trying rkt demo under TinyCore linux (kernel 3.16)

I am running TinyCore v6.0, CorePure64 version with kernel version 3.16.6-tinycore64.  TinyCore is itself running as a guest inside a KVM host.
This issue is being raised after discussion with iaguis on the #rkt irc channel.
When I try and run a basic rkt demo (loading the basic alpine-sh container) I am getting read-only errors, as per this debug output:
/mnt/vda1/dd/rkt-v1.1.0$ sudo ./rkt run --interactive quay.io/coreos/alpine-sh --debug
image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.1.0
image: using image from local store for image name quay.io/coreos/alpine-sh
stage0: Preparing stage1
stage0: Writing image manifest
stage0: Loading image sha512-a2fb8f390702d3d9b00d2ebd93e7dd1c241ee7bc752b362635c57b29aabb1a48
stage0: Writing image manifest
stage0: Writing pod manifest
stage0: Setting up stage1
stage0: Wrote filesystem to /var/lib/rkt/pods/run/1b359f08-3b11-4e4d-9701-7f95149e7cff
stage0: Pivoting to filesystem /var/lib/rkt/pods/run/1b359f08-3b11-4e4d-9701-7f95149e7cff
stage0: Execing /init
networking: loading networks from /etc/rkt/net.d
networking: loading network default with type ptp
stage1: warning: error setting journal ACLs, you'll need root to read the pod journal
  └─error setting default ACLs on "stage1/rootfs/var/log/journal"
    └─error calling acl_set_file
      └─operation not supported
stage1: continuing with per-app isolators disabled
  └─could not determine if we're running from a unit file
    └─unable to open a handle to libsystemd
Spawning container rkt-1b359f08-3b11-4e4d-9701-7f95149e7cff on /var/lib/rkt/pods/run/1b359f08-3b11-4e4d-9701-7f95149e7cff/stage1/rootfs.
Press ^] three times within 1s to kill container.
Failed to create directory /var/lib/rkt/pods/run/1b359f08-3b11-4e4d-9701-7f95149e7cff/stage1/rootfs/sys/fs/selinux: Read-only file system
Failed to create directory /var/lib/rkt/pods/run/1b359f08-3b11-4e4d-9701-7f95149e7cff/stage1/rootfs/sys/fs/selinux: Read-only file system
/etc/localtime is not a symlink, not updating container timezone.
systemd 222 running in system mode. (-PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
Detected virtualization systemd-nspawn.
Detected architecture x86-64.
Running with unpopulated /etc.

Welcome to Linux!

Initializing machine ID from container UUID.
Populated /etc with preset unit settings.
Failed to install release agent, ignoring: Read-only file system
Failed to create root cgroup hierarchy: Read-only file system
Failed to allocate manager object: Read-only file system
[!!!!!!] Failed to allocate manager object, freezing.

cat /proc/cgroups output is below...
$ cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  0       1       1
cpu     0       1       1
cpuacct 0       1       1
devices 0       1       1
freezer 0       1       1
net_cls 0       1       1
blkio   0       1       1
perf_event      0       1       1
net_prio        0       1       1

cat /proc/1/mountinfo output is below (with the tcloop squashfs mounts 16-27 removed for simplicity):
/mnt/vda1/dd/rkt-v1.1.0$ cat /proc/1/mountinfo
1 1 0:2 / / rw - rootfs rootfs rw,size=1851300k,nr_inodes=678308
10 1 0:4 / /proc rw,relatime - proc proc rw
11 1 0:11 / /sys rw,relatime - sysfs sysfs rw
12 1 0:9 / /dev/pts rw,relatime - devpts devpts rw,mode=600,ptmxmode=000
13 1 0:12 / /dev/shm rw,relatime - tmpfs tmpfs rw
14 11 0:13 / /sys/fs/fuse/connections rw,relatime - fusectl fusectl rw
15 1 250:1 / /mnt/vda1 rw,noatime - ext4 /dev/vda1 rw,data=ordered
<tcloop mounts snipped>
28 1 0:4 / /var/lib/rkt/pods/run/a8f6c36f-9ea6-45cb-a2af-99ee57376afb/netns rw,relatime - proc proc rw
29 1 0:4 / /var/lib/rkt/pods/run/e02e0f44-044a-4f32-a164-0a23b4437540/netns rw,relatime - proc proc rw
30 1 0:4 / /var/lib/rkt/pods/run/190f0ba6-000b-4e9e-8f56-2723a0f8fd3d/netns rw,relatime - proc proc rw
31 1 0:4 / /var/lib/rkt/pods/run/e1aa7851-7312-4e11-8587-c762f863cd07/netns rw,relatime - proc proc rw
32 1 0:4 / /var/lib/rkt/pods/run/ac8500b2-92f6-4208-9e2b-ad18b6d246e8/netns rw,relatime - proc proc rw
33 1 0:4 / /var/lib/rkt/pods/run/1b359f08-3b11-4e4d-9701-7f95149e7cff/netns rw,relatime - proc proc rw

iaguis noted that there are no cgroups mounted in the above output.  However, please note that tinycore uses init, and not systemd (apparently cgroup mounting is a systemd thing).
Any ideas on how to resolve this would be excellent!  Rumour is that using rkt in this situation (running rkt on a linux kernel 3.17 guest inside a KVM/other host) should be possible... but if not, that would also be good to know.

To fully replicate this problem, assuming you have qemu available, this sequence does it:
Get the latest TinyCore iso and launch it with qemu with 1GB RAM:
curl -LO http://tinycorelinux.net/7.x/x86_64/release/CorePure64-7.0.iso
qemu-system-x86_64 -m 1024 -curses -cdrom CorePure64-7.0.iso

Press enter inside to boot the system.  Then paste in the following commands to set up an environment to run rkt with, and make the (failing) attempt...
# install required packages...
tce-load -wi curl acl iptables
# download rkt...
curl -LO https://github.com/coreos/rkt/releases/download/v1.1.0/rkt-v1.1.0.tar.gz
tar zxf rkt*
cd rkt*
# set up the rkt group and membership...
sudo addgroup rkt
sudo adduser tc rkt
# Fix a tinycore oddity...
sudo ln -s /lib /lib64
# Try and run the alpine-sh container...
sudo ./rkt run --interactive quay.io/coreos/alpine-sh --debug

After confirming to trust the key with "yes", you should then see the error, as in this image:
http://i.snag.gy/zCKwL.jpg
I was unable to figure out how to send ^] three times to qemu, so you can get locked up there.  To get out, hit Esc+2, and then type quit at the (qemu) prompt.
Note that TinyCore is running entirely in RAM here.

The problem here seems to be that TinyCore doesn't mount /sys/fs/cgroup which systemd expects to be mounted.
Then, nspawn remounts /sys read-only, and tries to do some operations in /sys/fs/cgroup  but fails because it is RO.

I tried sudo mount -t tmpfs cgroup_root /sys/fs/cgroup first, but this did not seem to be enough.  I get the same errors.
Also - you said "which systemd expects to be mounted".  Does rkt require systemd?  Will it not work on init system?
Is there something else that is wrong?  Maybe permissions on that cgroup mount?
tc@box:~$ sudo mount -t tmpfs cgroup_root /sys/fs/cgroup
tc@box:~$ mount | grep cgroup
cgroup_root on /sys/fs/cgroup type tmpfs (rw,relatime)
tc@box:~$ ls -al /sys/fs
total 0
drwxr-xr-x    6 root     root             0 Apr 14 16:31 ./
dr-xr-xr-x   13 root     root             0 Apr 14 16:31 ../
drwxrwxrwt    2 root     root            40 Apr 14 16:31 cgroup/

Following my debian setup (where rkt works) I see that the cgroup_root is mounted with mode=755.  I tried this but it didn't change anything (same "Read-only file system" errors):
tc@box:~$ sudo mount -t tmpfs -o mode=755 cgroup_root /sys/fs/cgroup
tc@box:~$ mount | grep cgroup
cgroup_root on /sys/fs/cgroup type tmpfs (rw,relatime,mode=755)

Just to be complete, I noticed my debian mount was 'cgroup' and not 'cgroup**_root**', so I tried that in TinyCore... but to no avail.  Same problems.
tc@box:~$ sudo mount -t tmpfs -o mode=755 cgroup /sys/fs/cgroup
tc@box:~$ mount | grep cgroup
cgroup on /sys/fs/cgroup type tmpfs (rw,relatime,mode=755)

I tried a different cgroup setup to no avail.  Specifically, I set cgroup up by adding:
cgroup  /sys/fs/cgroup  cgroup  defaults  0   0

to /etc/fstab and mounting it.  This seemed to kick off cgroups ok:
tc@box:~$ sudo mount /sys/fs/cgroup
tc@box:~$ mount | grep cgroup
cgroup on /sys/fs/cgroup type cgroup (rw,relatime,cpuset,cpu,cpuacct,blkio,devic
es,freezer,net_cls,perf_event,net_prio)
tc@box:~$ ls -a /sys/fs/cgroup
./                               cpuset.memory_migrate
../                              cpuset.memory_pressure
blkio.reset_stats                cpuset.memory_pressure_enabled
cgroup.clone_children            cpuset.memory_spread_page
cgroup.procs                     cpuset.memory_spread_slab
cgroup.sane_behavior             cpuset.mems
cpu.rt_period_us                 cpuset.sched_load_balance
cpu.rt_runtime_us                cpuset.sched_relax_domain_level
cpu.shares                       devices.allow
cpuacct.stat                     devices.deny
cpuacct.usage                    devices.list
cpuacct.usage_percpu             net_cls.classid
cpuset.cpu_exclusive             net_prio.ifpriomap
cpuset.cpus                      net_prio.prioidx
cpuset.effective_cpus            notify_on_release
cpuset.effective_mems            release_agent
cpuset.mem_exclusive             tasks
cpuset.mem_hardwall

... however, the sample Alpine linux container still failed to start:

Also of note in that screencap is that it really looks like rkt is wanting/requiring systemd. Still not clear if systemd is a requirement?  Rkt docs say it will "easily integrate with existing init systems, like systemd", but that doesn't say "needs systemd"...
If me trying to get rkt working on an init system is futile, please just let me know! :)  Or maybe it is that my testbed for this inside qemu is a bad idea that will never work?

It definitely shouldn't depend on systemd on the host. Sorry, we haven't had a chance to look into your issue. But getting it working on another system is definitely not futile, we'd be very glad if you could figure this out :-)

Could the root cause here be that the available cgroups on the host system aren't sufficient?  Here's what I get when I try and mount them all (note the conspicuous absence of memory, for example):
tc@box:~$ sudo mkdir /cgroups
tc@box:~$ sudo mount -t cgroup none /cgroups
tc@box:~$ grep cgroup /proc/mounts
none /cgroups cgroup rw,relatime,cpuset,cpu,cpuacct,blkio,devices,
freezer,net_cls,perf_event,net_prio 0 0
tc@box:~$
tc@box:~$ ls /cgroups
blkio.reset_stats                cpuset.memory_pressure
cgroup.clone_children            cpuset.memory_pressure_enabled
cgroup.procs                     cpuset.memory_spread_page
cgroup.sane_behavior             cpuset.memory_spread_slab
cpu.rt_period_us                 cpuset.mems
cpu.rt_runtime_us                cpuset.sched_load_balance
cpu.shares                       cpuset.sched_relax_domain_level
cpuacct.stat                     devices.allow
cpuacct.usage                    devices.deny
cpuacct.usage_percpu             devices.list
cpuset.cpu_exclusive             net_cls.classid
cpuset.cpus                      net_prio.ifpriomap
cpuset.effective_cpus            net_prio.prioidx
cpuset.effective_mems            notify_on_release
cpuset.mem_exclusive             release_agent
cpuset.mem_hardwall              tasks
cpuset.memory_migrate

Kernel in this case is a 4.2.9 build:
tc@box:~$ uname -a
Linux box 4.2.9-tinycore64 #1999 SMP Mon Jan 18 19:59:34 UTC 2016 x86_64 GNU/Linux

It might be, but I'd expect different errors rather than it still complaining about a read-only filesystem.. @iaguis  any ideas?

I just tried http://tinycorelinux.net/7.x/x86_64/release/TinyCorePure64-7.0.iso with rkt-v1.6.0 and it seems to work:
tc@box:~/rkt-v1.6.0$ uname -a
Linux box 4.2.9-tinycore64 #1999 SMP Mon Jan 18 19:59:34 UTC 2016 x86_64 GNU/Linux
tc@box:~/rkt-v1.6.0$ sudo ./rkt run quay.io/coreos/alpine-sh --exec echo -- HELLO WORLD
image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.6.0
image: using image from local store for image name quay.io/coreos/alpine-sh
networking: loading networks from /etc/rkt/net.d
networking: loading network default with type ptp
stage1: warning: error setting journal ACLs, you'll need root to read the pod journal: operation not supported
stage1: continuing with per-app isolators disabled: unable to open a handle to libsystemd
echo[5]: HELLO WORLD
tc@box:~/rkt-v1.6.0$

Confirmed!  Great! I just tried rkt 1.6 and it works.
I'm actually a bit surprised that it did work. I've been educating myself on cgroups and it turns out that the tinycore kernel doesn't have the cgroup memory subsystem compiled in. I thought this might have been the obstacle and have been working on a new kernel build to enable it.

rkt doesn't require the memory cgroup subsystem. If it's not there, the memory isolator won't work but the rest should work.
Actually, it seems per-app isolators won't work if libsystemd is not present on the host, we should fix that.

@iaguis close this and file follow up ..?

https://github.com/coreos/rkt/issues/2670