should.not.equal(complexObject) is not optimal

Assuming we have two complex objects, such as registry1 and registry2, the following assertion becomes very expensive:
registry1.should.not.equal(registry2)
should at the moment always asserts the equal first, creates an AssertionError, and then tests the not condition by catching the error. See https://github.com/shouldjs/should.js/blob/master/lib/assertion.js#L47-L68. For complex objects, the getMessage is very expensive (in our case, more than 4 seconds).
The work around is to provide a custom message for the assertion. I wonder if the assert could check the negate flag and only construct a dummy AssertionError if negate is true?
See https://github.com/strongloop/loopback-datasource-juggler/pull/431

I understand the issue, If it is not hard is it possible to make for me some example that i can add to tests, while i am working on fix.

Sure, I'll add test case.

Here we go:
it('should optimize not.equal for complex objects', function() {
    function mockComplexObject(prefix, depth, width) {
      if (depth <= 0) {
        return 'value-' + prefix;
      }
      var obj = {};
      for (var i = 0; i < width; i++) {
        obj[prefix + depth + '_' + i] = mockComplexObject(prefix, depth-1, width);
      }
      return obj;
    }
    var a = mockComplexObject('a', 8, 4);
    var b = mockComplexObject('b', 8, 4);

    a.should.be.not.equal(b); // It takes a while
  });

Please, try 4.6.5. For nested assertions it will not generate messages while not need. Looks not ideal, will refactor this later, but should fix your issue.
Let me know if that works for you.

@btd Thanks for the fix!


An unhandled exception occurred: schema with key or id ... already exists

🐞 Bug report
Command (mark with an x)
(all, bare ng command or things like ng new)
Is this a regression?
Yes
Description
When I run ng installed globally it doesn't work
🔬 Minimal Reproduction
This may be specific to my environment, but I'm using

Node 12.18.0
<EMAIL_ADDRESS>
🔥 Exception or Error

$ ng
An unhandled exception occurred: schema with key or id "http://json-schema.org/draft-06/schema" already exists
See "/tmp/ng-8tc5cm/angular-errors.log" for further details.

The error log shows this:
[error] Error: schema with key or id "http://json-schema.org/draft-06/schema" already exists
    at checkUnique (/home/stephen/.config/yarn/global/node_modules/ajv/lib/ajv.js:477:11)
    at Ajv.addSchema (/home/stephen/.config/yarn/global/node_modules/ajv/lib/ajv.js:140:3)
    at Ajv.addMetaSchema (/home/stephen/.config/yarn/global/node_modules/ajv/lib/ajv.js:156:8)
    at new CoreSchemaRegistry (/home/stephen/.config/yarn/global/node_modules/@angular/cli/node_modules/@angular-devkit/core/src/json/schema/registry.js:67:19)
    at Object.runCommand (/home/stephen/.config/yarn/global/node_modules/@angular/cli/models/command-runner.js:92:22)
    at default_1 (/home/stephen/.config/yarn/global/node_modules/@angular/cli/lib/cli/index.js:62:54)
    at /home/stephen/.config/yarn/global/node_modules/@angular/cli/lib/init.js:141:12

🌍 Your Environment

yarn 1.22.4
nodejs 12.18.0
angular/cli 10.0.0-rc.2

I'd be happy to give access to my environment via video call if that would help.

Most likely something went wrong during the installation.
Can you try to uninstall and install @angular/cli again?

Purged global node modules and reinstalled, worked. Wonder what went wrong... Oh well, thanks!


cpplint not running on Atom 0.149 and 0.150

Not getting any output from the linter with C++ files. linter-flake8 is working with Python files, so not sure what the issue is. Happy to help debug, but not sure where to start.

Marking as closed, the plugin has been re-written since this was filed, and it looks like a fix was implemented anyway.


empty lines in code cause null pointer exception

Title is self-explanatory.

Ok we try to fix that as soon as possible


feat measurement visibility

Includes

Visibility icon at measurement panel
Visibility functionality
Hotkeys assigned for toggling visibility: m for current active measurement, shift+m for all measurements

PR Checklist

[ ] Brief description of changes
[ ] Links to any relevant issues
[ ] Required status checks are passing
[ ] User cases if changes impact the user's experience
[ ] @mention a maintainer to request a review

Generally the changes are mostly ok.  A couple of things that I think would make it a bit more useful:

Remove the icon from the table to toggle visibility.  Having too many icons will be a problem.  If we can figure out how to make right-click in the table bring up the context menu, then that would be useful.
If the user clicks the measurement for any reason, make it visible
Have a global hide-measurements command, acting on all objects (similarly a global show all)

I think the user experience is better that way.  We can talk tomorrow about how it works.

Generally the changes are mostly ok. A couple of things that I think would make it a bit more useful:

Remove the icon from the table to toggle visibility.  Having too many icons will be a problem.  If we can figure out how to make right-click in the table bring up the context menu, then that would be useful.
If the user clicks the measurement for any reason, make it visible
Have a global hide-measurements command, acting on all objects (similarly a global show all)

I think the user experience is better that way. We can talk tomorrow about how it works.

We definitely can make it, shall we? I dont disagree but as a user I would prefer to have the icon already displayed instead of having to do two actions to hide it (one to open menu other to hide/show)
Measurement you mean on viewport? or on the table? If first one is the case: This is really doable, but, the solution should go under image library, otherwise it will be a workaround. Shall we? I can not estimate the effort yet. What about second one: what should happen if user clicks on measurement item (on table)?
Alright, and will there be a icon for that? Where to place it.

@wayfarer3130  code is updated with what we discussed internally. Can you take a look please?

Do you need this? If so can you please rebase on v3-stable?

will revisit this here
https://github.com/OHIF/Viewers/issues/4095


Revert breaking change in enum deserialization

There appears to have been a change right around Dropwizard 1.0
in how enums are deserialized; my guess is that somehow @JsonCreator
cascades and prevents the FuzzyEnumModule from being applied properly,
but I'm not really sure. The result was that SingularityContainerType
wasn't accepting lowercase mesos, docker as valid values, which was
a breaking change from before - and in particular, prevented creating
deploys from the UI.
See also dropwizard/dropwizard#1985, which is kind-of related, I think
/cc @ssalinas

@PtrTeixeira I can't seem to reproduce this one. I tried the same fields via ui in our staging setup and a lower case docker was correctly read as the SingularityContainerType.DOCKER. Same with a number of other fields. Going to close this one for the moment, if you have more specific examples of failures feel free to reopen


Update to ungoogled-chromium 105.0.5195.102-1

No patch updates required. I am still waiting for the compiling to finish, but I am very optimistic that it will succeed. As always, I would like to wait for the compiling to finish (not necessarily on my setup if anyone has a faster setup) before we merge.

FYI my compile suceeded.
I have a 5950x with 64GBs of ram and a Gen4 SSD so I can probably compile faster than most.  Feel free to ping me if you'd like me to keep you informed as to my compile successes (or lack thereof).

@R-T-B Thanks for the feedback, I'll keep you in mind in case I need results faster :)
Meanwhile compiling finished on my setup as well:

ungoogled-chromium-windows was never so alive and kicking, thanks guys!


raft: use configured logger in raft/node.go

Those three log statements in node.go have not been using the logger that was passed via raft.Config, but instead the default raft logger. This changes it to use the proper logger.

LGTM. Thanks for fixing this!


Fix ssz publishBlock sent without milestone header

PR Description
It was reported that VC Teku fails to propose in connection with BN Lighthouse.
This happens because Teku sends requests without required header (see https://github.com/ethereum/beacon-APIs/blob/master/apis/beacon/blocks/blocks.yaml#L37 "Use content type header"). According to the API spec it should be used both for blind block and non-blind. Fixed with the PR.
Tested also (successfully) in Kurtosis with following configuration:
participants:
  - el_client_type: geth
    cl_type: lighthouse
    vc_type: teku
    vc_image: consensys/teku:develop
additional_services:
  - el_forkmon
  - tx_spammer
  - dora
snooper_enabled: true

Fixed Issue(s)

Documentation

[x] I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

[ ] I thought about adding a changelog entry, and added one if I deemed necessary.

this looks like we're still using the v1 block interface? I guess at some point this starts calling v3 and it needs that header too...


read func return after read success

It has no need to wait for the p is full filled to return?

This has been implemented in the main branch.


Responder

Large packets for response messages that can aggregate data

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@4d7d4d6). Click here to learn what that means.
The diff coverage is 86.2%.

@@            Coverage Diff            @@
##             master      #77   +/-   ##
=========================================
  Coverage          ?   92.75%
=========================================
  Files             ?       15
  Lines             ?     1159
  Branches          ?        0
=========================================
  Hits              ?     1075
  Misses            ?       84
  Partials          ?        0

Impacted Files
Coverage Δ

src/result.rs
89.04% <ø> (ø)

src/streamer.rs
87.08% <85%> (ø)

src/accountant_skel.rs
89.09% <88.88%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d7d4d6...290960c. Read the comment docs.

LGTM

@garious any idea why this diff shows up in CI, but not locally?
https://travis-ci.org/loomprotocol/silk/jobs/358193740#L460


fork-ts-checker-webpack-plugin not called with --max-old-space from NODE_OPTIONS

Describe the bug
The fork-ts-checker-webpack-plugin is not called with --max-old-space from NODE_OPTIONS.
Did you try recovering your dependencies?
$ npm --version
6.14.8
$ node --version
v12.19.1
$ cat package.json | grep \"react-scripts\"
    "react-scripts": "~3.4.3"
$ cat package-lock.json | grep -A 2 \"react-scripts\"
    "react-scripts": {
      "version": "3.4.4",
      "resolved": "https://registry.npmjs.org/react-scripts/-/react-scripts-3.4.4.tgz",

Environment
FROM node:12.19.1-alpine3.12
PID   USER     TIME  COMMAND
    1 root      0:00 {node} npm
   16 root      0:00 node /opt/app/node_modules/.bin/react-scripts build
   23 root      1:18 node /opt/app/node_modules/react-scripts/scripts/build.js
   34 root      0:51 /usr/local/bin/node --max-old-space-size=2048 /opt/app/node_modules/fork-ts-checker-webpack-plugin/lib/service.js
   45 root      0:00 /bin/sh
   54 root      0:00 ps
/opt/app # cat /proc/34/environ | tr '\0' '\n' | grep NODE_OPTIONS
NODE_OPTIONS=--max-old-space-size=4096
/opt/app # cat .env.production | grep NODE_OPTIONS
NODE_OPTIONS=--max-old-space-size=4096

Steps to reproduce
.env.production
NODE_OPTIONS=--max-old-space-size=4096
npm run build inside node:12.19.1-alpine3.12 docker image.
Expected behavior
The configured max-old-space-size value from NODE_OPTIONS is used.
Actual behavior
--max-old-space-size=2048 is always used.
Which leads to:
<--- Last few GCs --->

[23:0x55822a148e00]   183795 ms: Mark-sweep 987.2 (995.5) -> 986.4 (995.5) MB, 804.3 / 0.2 ms  (average mu = 0.210, current mu = 0.191) allocation failure scavenge might not succeed
[23:0x55822a148e00]   185184 ms: Mark-sweep 987.0 (995.5) -> 986.5 (995.8) MB, 1249.9 / 0.1 ms  (average mu = 0.148, current mu = 0.100) allocation failure GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x558228289b39]
Security context: 0x25445edc08d1 <JSObject>
    1: SourceNode_walk [0x22ffba46da21] [/opt/app/node_modules/webpack-sources/node_modules/source-map/lib/source-node.js:~221] [pc=0x3d2731fba85](this=0x31ad9ffd6409 <SourceNode map = 0x3d40622bbfd9>,0x1b75d4f9e431 <JSFunction (sfi = 0x348957ffd339)>)
    2: SourceNode_walk [0x22ffba46da21] [/opt/app/node_modules/webpack-sources/node_modules/source-map/lib/...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
npm ERR! code ELIFECYCLE
npm ERR! errno 1

We have also the same error after setting the variable with export NODE_OPTIONS=--max-old-space-size=6144


Run tests in CI

We have a tests folder, would be nice to run them on github actions

@sepfy any reason most of the tests are disabled?

The test directory contains simple test programs I developed during my own testing, but I haven't had time to integrate them into the GitHub Actions workflow yet (Including automated verification of whether the function's output is correct, and so on.), so I have temporarily disabled it.

Ok I will work on adding tests then!
 Hopefully prevents some crashes in the future :)


Add LRU cache with expirable keys

Reactivated Caching with LRU Cache with expirable Keys; Added Logs and Metrics

Re-enabled caching for image architectures.
Switched to an LRU (Least Recently Used) cache with a maximum of 256 entries and expire after 6 hours to ensure freshness.
Added logging around cache operations, including cache hits, misses.
Added metrics to track cache usage, time taken to process cache hits/misses with Histograms, and cache size with Gauge.
Implemented integration tests to validate caching functionality.

nit: cashe -> cache

/retest-required

/retest-required

/test ocp417-e2e-gcp

/lgtm
/approve
Thanks @AnnaZivkovic

@lwan-wanglin note that we missed an optional: true in the 4.17 test config and it is not set as required.

@lwan-wanglin note that we missed an optional: true in the 4.17 test config and it is not set as required.
fixed in pr https://github.com/openshift/release/pull/57201, PTAL, thanks!

/retest

/retest

/unhold
/lgtm
/approve

We need to open a new story for the polling goroutine to expose the size of the cache as it is exportable and the size can change regardless of the cache being used by the GetCompatibleArchitectureSet method

/test  'Red Hat Konflux / multiarch-tuning-operator-enterprise-contract / multiarch-tuning-operator'

/test  Red Hat Konflux/Red Hat Konflux/multiarch-tuning-operator-enterprise-contract/multiarch-tuning-operator

/retest-required

/override "Red Hat Konflux/Red Hat Konflux/multiarch-tuning-operator-enterprise-contract/multiarch-tuning-operator"
The failure is due to another issue in the base image. We only need to wait for this to eventually pass.
Overriding to make this land

/override "Red Hat Konflux/Red Hat Konflux/multiarch-tuning-operator-enterprise-contract/multiarch-tuning-operator"

/override "Red Hat Konflux / multiarch-tuning-operator-enterprise-contract / multiarch-tuning-operator"


[BUG] Unable to restore latest backup if multiple backups exist.

Describe the bug
A volume that is restored from the latest backup of a volume having multiple backups fails to attach to a pod.
To Reproduce
Steps to reproduce the behavior:

Create a volume, Write data to it file1.
Take a backup b1.
Write more data into it file2 and take backup b2.
Restore the latest backup b2.
Wait for the restore to complete.
Create PV/PVC and attach it to a pod.
The pod doesn't come up healthy and shows below error.

MountVolume.SetUp failed for volume "restore-2-vol-1" : rpc error: code = Internal desc = 'fsck' found errors on device /dev/longhorn/restore-2-vol-1 but could not correct them: fsck from util-linux 2.31.1 /dev/longhorn/restore-2-vol-1: recovering journal /dev/longhorn/restore-2-vol-1: Superblock has an invalid journal (inode 8). CLEARED. *** journal has been deleted *** /dev/longhorn/restore-2-vol-1: Resize inode not valid. /dev/longhorn/restore-2-vol-1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options)

Expected behavior
Pod should come up healthy and should have the data file1 and file2
Log
longhorn-support-bundle_96ca2f0c-2c0b-4a51-8f35-56a5f6b0a6a4_2020-09-01T00-42-25Z.zip
Pod restore-2, restore-2-vol-2 fail to come healthy but in longhorn UI volume restore-2-vol-1, restore-2-vol-2 show as attached to these pods.
Environment:

Longhorn version: Master - 08/31/2020
Kubernetes version: 18.6
Node OS type and version: Ubuntu 18.04

Additional context:
This works fine with longhorn v1.0.2.

Not sure if it's related to the recent backupstore changes, @joshimoo can you take a look?

Will investigate.

@yasker @joshimoo Looks like this problem is on the restore side. Also, the occurrence of the problem is often not always.
I created multiple backups with Longhorn Master and tried restoring the same backup in both Longhorn Master and Longhorn v1.0.2.

With v1.0.2, restore is always successful and restored volume can be attached to the pod.
With Master, restore volume shows Actual size as 0 sometimes after the restore completes and the attched pod fails to come up healthy.

logs:
time="2020-09-01T18:31:46Z" level=info msg="Override size of volume restore-5 to<PHONE_NUMBER> because it's from backup"
time="2020-09-01T18:31:46Z" level=debug msg="Created volume restore-5: {Size:2147483648 Frontend:blockdev FromBackup:s3://khushboo-longhorn@us-west-1/?backup=backup-e8dcf4c688444324&volume=volume-test-1 NumberOfReplicas:3 DataLocality:disabled StaleReplicaTimeout:20 NodeID: EngineImage:longhornio/longhorn-engine:master RecurringJobs:[] BaseImage: Standby:false DiskSelector:[] NodeSelector:[] DisableFrontend:false}"
<IP_ADDRESS> - - [01/Sep/2020:18:31:45 +0000] "POST /v1/volumes HTTP/1.1" 200 970 "https://<IP_ADDRESS>/k8s/clusters/c-fwrr8/api/v1/namespaces/longhorn-system/services/http:longhorn-frontend:80/proxy/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
<IP_ADDRESS> - - [01/Sep/2020:18:31:48 +0000] "GET /v1/volumes/restore-4 HTTP/1.1" 200 5342 "" "Go-http-client/1.1"
<IP_ADDRESS> - - [01/Sep/2020:18:31:48 +0000] "GET /v1/settings/mkfs-ext4-parameters HTTP/1.1" 200 590 "" "Go-http-client/1.1"
time="2020-09-01T18:31:49Z" level=debug msg="Replica controller khush-test-lh-wk2 picked up restore-5-r-41841ce0"
time="2020-09-01T18:31:49Z" level=debug msg="Instance process restore-5-r-41841ce0 had been created, need to wait for instance manager update"
time="2020-09-01T18:31:49Z" level=debug msg="Instance process restore-5-r-41841ce0 had been created, need to wait for instance manager update"
time="2020-09-01T18:31:50Z" level=debug msg="Instance restore-5-r-41841ce0 starts running, IP <IP_ADDRESS>"
time="2020-09-01T18:31:50Z" level=debug msg="Instance restore-5-r-41841ce0 starts running, Port 10075"
time="2020-09-01T18:31:50Z" level=debug msg="Instance handler updated instance restore-5-r-41841ce0 state, old state stopped, new state running"
<IP_ADDRESS> - - [01/Sep/2020:18:32:53 +0000] "GET /v1/volumes/restore-4 HTTP/1.1" 200 5342 "" "Go-http-client/1.1"
<IP_ADDRESS> - - [01/Sep/2020:18:32:53 +0000] "GET /v1/settings/mkfs-ext4-parameters HTTP/1.1" 200 590 "" "Go-http-client/1.1"
time="2020-09-01T18:33:08Z" level=debug msg="Prepare to delete instance restore-5-r-41841ce0"
time="2020-09-01T18:33:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"restore-5-r-41841ce0\", UID:\"72c3fa5d-7061-48e2-8d5b-06d08a3f7be1\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"27948\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops restore-5-r-41841ce0"
time="2020-09-01T18:33:08Z" level=debug msg="Prepare to delete instance restore-5-r-41841ce0"
time="2020-09-01T18:33:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"restore-5-r-41841ce0\", UID:\"72c3fa5d-7061-48e2-8d5b-06d08a3f7be1\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"27950\", FieldPath:\"\"}): type: 'Normal' reason: 'Stop' Stops restore-5-r-41841ce0"
time="2020-09-01T18:33:08Z" level=debug msg="Instance handler updated instance restore-5-r-41841ce0 state, old state running, new state stopped"
time="2020-09-01T18:33:54Z" level=debug msg="websocket: close 1001 (going away)" id=2283688071133541039 type=nodes
time="2020-09-01T18:33:54Z" level=debug msg="websocket: close 1001 (going away)" id=4110104903864334076 type=volumes
<IP_ADDRESS> - - [01/Sep/2020:17:56:02 +0000] "GET /v1/ws/1s/engineimages HTTP/1.1" 200 0 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
time="2020-09-01T18:33:54Z" level=debug msg="websocket: close 1001 (going away)" id=4626667678681971331 type=engineimages
time="2020-09-01T18:33:54Z" level=debug msg="websocket: close 1001 (going away)" id=2881063850674186457 type=settings
time="2020-09-01T18:33:54Z" level=debug msg="websocket: close 1001 (going away)" id=8308911426065941244 type=events
<IP_ADDRESS> - - [01/Sep/2020:17:56:07 +0000] "GET /v1/ws/1s/events HTTP/1.1" 200 0 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
time="2020-09-01T18:33:54Z" level=warning msg="HTTP handling error websocket: close sent"
time="2020-09-01T18:33:54Z" level=error msg="Error in request: websocket: close sent"
2020/09/01 18:33:54 http: response.WriteHeader on hijacked connection from github.com/gorilla/handlers.(*responseLogger).WriteHeader (handlers.go:65)
2020/09/01 18:33:54 http: response.Write on hijacked connection from github.com/gorilla/handlers.(*responseLogger).Write (handlers.go:59)
time="2020-09-01T18:33:54Z" level=error msg="Failed to write err: websocket: close sent"

@khushboo-rancher Currently the detached volume doesn't have actualSize field. @smallteeths is working on adding it.

Verified on Master - 09/03/2020
After revert, the reported problem is not reproducible.


Reference to Aggregator in UML / call flow

Problem description
We should refer to an aggregator in UML / call flow description
Expected action
We discussed that a reference should be made to documentation in identity & consent group.
Additional context

Please refer to #71
diagram source files


Milestone9: Add devise

In this milestone we:

Installed and setup devise
Implemented the following

Can register a new user.
User logs in with a combination of email and password.
Hashed password should be stored in the database.
Ask for confirmation of email.
Can reset password.

thank you


Hello, how is the effect of the music generated by this project ai

Hello, how is the effect of the music generated by this project ai

@xuboot I am sorry, but I do not understand your question.

@asigalov61 How about this project, what is the effect of the music generated by ai, how can I test this project of yours, or where can I get the data myself and run it to see the final result?

@xuboot You can listen to output samples from all projects here: https://soundcloud.com/aleksandr-sigalov-61/albums
You can start with Los Angeles Music Composer: https://github.com/asigalov61/Los-Angeles-Music-Composer
Demo Google Colab with the model is located here: https://colab.research.google.com/github/asigalov61/Los-Angeles-Music-Composer/blob/main/Los_Angeles_Music_Composer.ipynb
Data/Dataset maker is here: https://github.com/asigalov61/Los-Angeles-Music-Composer/blob/main/Training-Data/Los_Angeles_Music_Composer_Training_Dataset_Maker.ipynb
And training code is here: https://github.com/asigalov61/Los-Angeles-Music-Composer/blob/main/Training-Code/Los_Angeles_Music_Composer_Maker.ipynb
Hope this helps and answers your questions.
Thank you.
Alex

@asigalov61 Okay, thanks, I would like to ask, which thesis technology used in this, at the same time, I see that the code does not generate midi files, in fact, I just want to generate songs through ai, while the last can change the midi file, which is currently the most used open source model on the market, or have a ready-made github address not

@asigalov61 Listened to it, felt good, but it seems I did not use to see the use of the model used, generally I want to focus on the input and output, the input is a midi file, the output is also a midi file

@asigalov61 There is also that this is geared towards the generation of ai music, right, not just the generation of piano music, right?

@xuboot Los Angeles Music Composer takes MIDI as input and generates MIDI as output. It is a multi-instrumental model (12 instruments) so you can select any available instrument.
With Los Angeles Music Composer you can generate music, continue music, inpaint music, and harmonize melodies. It's all in MIDI format.
If you want audio, then you can try Riffusion or MuBerrt.
I hope this helps and answers your question.
You can try writing to me in your language too and I will try to translate it myself to better understand you (if you want).
Alex

@asigalov61 You guys do ai generate songs, model input is text or mid it, such as text is the type such as rock, and then the output is mid rock music, and then by uploading the mid output this kind of music similar to the mid it

@xuboot If you want text-to-music (text to MIDI), Los Angeles Music Composer will have that function soon. I will post the colab shortly and I will let you know.
If you want MIDI to MIDI, then use Los Angeles Music Composer demo colab. It supports custom MIDI continuation.
Alex.

@xuboot Try this: https://colab.research.google.com/github/asigalov61/Los-Angeles-Music-Composer/blob/main/Los_Angeles_Music_Composer_TTM_Edition.ipynb

@asigalov61 hello,Some do not know how I should test the running code without jupyter, because I want to test the entire workflow to facilitate learning, while the model calls encapsulated into a call to the interface to load the model online and provide services

@xuboot If you do not want to use Jupyter, then you will have to assemble the code yourself. Sorry about that.
Python versions are provided in the repo.
I.e. https://github.com/asigalov61/Los-Angeles-Music-Composer/blob/main/los_angeles_music_composer_ttm_edition.py

@asigalov61 Thank you, you have seen the demo, feel the effect of the generated music are poor, because the use of classical music training model, and then generate music, (this demo is also, https://github.com/netpi/compound-word-transformer-tensorflow), you can look at the reference a

@asigalov61 Hello, when you wrote this project, did you refer to Magenta, MuseGAN, Deepj project reference, or you

@asigalov61 Hello, are you referring to this paper to achieve
https://archives.ismir.net/ismir2021/paper/000017.pdf

Hello, please ask this project, what technical framework and thesis used to achieve, at the same time I found many parameters do not quite understand

@xuboot This is a control sequence generator. It tells the model the desired parameters for a generation.
These are guidance parameters/tokens that help the model generate the desired music.
intro_mode_time = statistics.mode([0] + [y[0] for y in melody_chords if y[2] != 9 and y[0] != 0])
intro_mode_dur = statistics.mode([y[1] for y in melody_chords if y[2] != 9])
intro_mode_pitch = statistics.mode([y[3] for y in melody_chords if y[2] != 9])
intro_mode_velocity = statistics.mode([y[4] for y in melody_chords if y[2] != 9])

@asigalov61 Hello that this project (https://github.com/asigalov61/Los-Angeles-Music-Composer) from MuseNet implementation well, or this project reference that paper implementation, I want to understand the concrete implementation of the principle and architecture, to facilitate my learning debugging code

2023131180590.zip
I tested it, input mid, output mid, the generated music is short, how do I control the length of the generated music, this file is a sample of my test, you can test it, I think the music you tested in soundcloud.com/aleksandr-sigalov-61/ inside is good

How do I reproduce the songs you generated on soundcloud.com/aleksandr-sigalov-61/ and find that the same mid file generates the same, I don't know if I didn't specify the type (such as guitar, rock, violin, etc., resulting in the same mid input, after the output mid, repeatedly the same mid I don't know where there is a parameter to control this piece, and then there is no detailed documentation)
Translated with www.DeepL.com/Translator (free version)

@asigalov61 How do I reproduce the songs you generated on soundcloud.com/aleksandr-sigalov-61/ and find that the same mid file generates the same, I don't know if I didn't specify the type (such as guitar, rock, violin, etc., resulting in the same mid input, after the output mid, repeatedly the same mid I don't know where there is a parameter to control this piece, and then there is no detailed documentation)

@xuboot You can control the length of the generated music by increasing the number of tokens to generate parameters.
If you want to specify instruments you will have to prime the model with the tokens of the instruments you want. There is no option to do that so you will have to write the code yourself.

@asigalov61 Actually, I was wondering how to modify the same mid to generate different kinds of mid, I saw in the code that there is (patch_map = [
[0, 1, 2, 3, 4, 5, 6, 7], # Piano
[24, 25, 26, 27, 28, 29, 30], # Guitar
[32, 33, 34, 35, 36, 37, 38, 39], # Bass
[40, 41], # Violin
[42, 43], # Cello
[46], # Harp
[56, 57, 58, 59, 60], # Trumpet
[64, 65, 66, 67, 68, 69, 70, 71], # Sax
[72, 73, 74, 75, 76, 77, 78], # Flute
[-1], # Drums
[52, 53], # Choir
[16, 17, 18, 19, 20] # Organ
]) do control the generation of different mid, but I do not know where to modify the parameters of the generation time, is to generate mid music, generated music and the original music as long as the code details where to modify it

@asigalov61 I don't quite understand this sentence (You can control the length of the generated music by increasing the number of tokens to generate parameters.), which line of this parameter code specifically controls the length of the generated music, and also, (If you want to specify instruments you will have to prime the model with the tokens of the instruments you want. There is no option to do that so you will have to There is no option to do that so you will have to write the code yourself.) I don't quite understand this sentence, which specific line of code to modify it, because the code does not have enough comments, it looks difficult to understand, I hope you can help me
这句话我不太懂（你可以通过增加生成参数的令牌数量来控制生成音乐的长度），这个参数代码的哪一行具体控制生成音乐的长度，而且，（如果你想指定工具，你必须用你想要的工具的标记来为模型做准备。没有选择这样做所以你必须去那里没有选择这样做所以你必须自己编写代码。)我不太明白这句话，具体要修改哪一行代码呢，因为代码没有足够的注释，看起来很难理解，希望你能帮我

@xuboot Sorry about the difficulty you are experiencing with my code....
Priming the model with instruments is done like this:
 #@title Improv Generator

#@markdown Select desired instruments (any combination is fine)

Piano = True #@param {type:"boolean"}
Guitar = True #@param {type:"boolean"}
Bass = True #@param {type:"boolean"}
Violin = False #@param {type:"boolean"}
Cello = False #@param {type:"boolean"}
Harp = False #@param {type:"boolean"}
Trumpet = False #@param {type:"boolean"}
Clarinet = False #@param {type:"boolean"}
Flute = False #@param {type:"boolean"}
Drums = True #@param {type:"boolean"}
Choir = False #@param {type:"boolean"}
Organ = False #@param {type:"boolean"}

#@markdown Improv Timings and Velocity
desired_prime_time = 10 #@param {type:"slider", min:0, max:127, step:1}
desired_prime_duration = 12 #@param {type:"slider", min:1, max:126, step:1}
desired_velocity = 6 #@param {type:"slider", min:1, max:8, step:1}

#@markdown Model settings

number_of_tokens_tp_generate = 512 #@param {type:"slider", min:32, max:4064, step:32}
number_of_batches_to_generate = 4 #@param {type:"slider", min:1, max:16, step:1}
temperature = 1 #@param {type:"slider", min:0.1, max:1, step:0.1}

instruments = []

if Piano:
  instruments += [
                  0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((0 * 128) + 60)+1152, # instrument number(1) + start pitch(60)
                 ]

if Guitar:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((1 * 128) + 60)+1152, # instrument number(1) + start pitch(60)
                  ]

if Bass:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((2 * 128) + 48)+1152, # instrument number(2) + start pitch(48)
                  ]
if Violin:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((3 * 128) + 72)+1152, # instrument number(3) + start pitch(72)
                  ]

if Cello:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((4 * 128) + 48)+1152, # instrument number(4) + start pitch(48)
                  ]

if Harp:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((5 * 128) + 72)+1152, # instrument number(5) + start pitch(72)
                  ]

if Trumpet:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((6 * 128) + 72)+1152, # instrument number(6) + start pitch(72)
                  ]

if Clarinet:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((7 * 128) + 72)+1152, # instrument number(7) + start pitch(72)
                  ]

if Flute:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((8 * 128) + 72)+1152, # instrument number(8) + start pitch(72)
                  ]

if Drums:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((9 * 128) + 35)+1152, # instrument number(9) + start pitch(35)
                  ]

if Choir:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((10 * 128) + 72)+1152, # instrument number(10) + start pitch(72)
                  ]

if Organ:
  instruments += [0, # delta start time (0)
                  ((desired_prime_duration * 8) + (desired_velocity-1))+128,
                  ((11 * 128) + 60)+1152, # instrument number(11) + start pitch(60)
                  ]

instruments[0] = desired_prime_time
instruments[3::3] = [0] * len(instruments[3::3])

outy = instruments

#===================================================================

print('=' * 70)
print('Los Angeles Music Composer Model Improvisation Generator')
print('=' * 70)

print('Generation settings:')
print('=' * 70)
print('Model temperature:', temperature)

print('=' * 70)
print('Selected Improv sequence:')
print(outy)
print('=' * 70)

inp = [outy] * number_of_batches_to_generate

inp = torch.LongTensor(inp).cuda()

#start_time = time()

out = model.module.generate(inp,
                      number_of_tokens_tp_generate,
                      temperature=temperature,
                      return_prime=True,
                      min_stop_token=0,
                      verbose=True)

out0 = out.tolist()

print('=' * 70)
print('Done!')
print('=' * 70)
#print('Generation took', time() - start_time, "seconds")
print('=' * 70)

#======================================================================

print('Rendering results...')
print('=' * 70)

for i in range(number_of_batches_to_generate):

  print('=' * 70)
  print('Batch #', i)
  print('=' * 70)

  out1 = out0[i]

  print('Sample INTs', out1[:12])
  print('=' * 70)

  if len(out) != 0:

      song = out1
      song_f = []
      tim = 0
      dur = 0
      vel = 0
      pitch = 0
      channel = 0

      son = []
      song1 = []

      for s in song:
        if s >= 128 and s < (12*128)+1152:
          son.append(s)
        else:
          if len(son) == 3:
            song1.append(son)
          son = []
          son.append(s)

      for ss in song1:

        tim += ss[0] * 10

        dur = (((ss[1]-128) // 8)+1) * 20
        vel = (((ss[1]-128) % 8)+1) * 15

        channel = (ss[2]-1152) // 128
        pitch = (ss[2]-1152) % 128

        song_f.append(['note', tim, dur, channel, pitch, vel ])

      detailed_stats = TMIDIX.Tegridy_SONG_to_MIDI_Converter(song_f,
                                                          output_signature = 'Los Angeles Music Composer',
                                                          output_file_name = '/content/Los-Angeles-Music-Composer-Music-Composition_'+str(i),
                                                          track_name='Project Los Angeles',
                                                          list_of_MIDI_patches=[0, 24, 32, 40, 42, 46, 56, 71, 73, 0, 53, 19, 0, 0, 0, 0],
                                                          number_of_ticks_per_quarter=500)

      print('=' * 70)
      print('Displaying resulting composition...')
      print('=' * 70)

      fname = '/content/Los-Angeles-Music-Composer-Music-Composition_'+str(i)

      x = []
      y =[]
      c = []

      colors = ['red', 'yellow', 'green', 'cyan', 'blue', 'pink', 'orange', 'purple', 'gray', 'white', 'gold', 'silver']

      for s in song_f:
        x.append(s[1] / 1000)
        y.append(s[4])
        c.append(colors[s[3]])

      FluidSynth("/usr/share/sounds/sf2/FluidR3_GM.sf2", 16000).midi_to_audio(str(fname + '.mid'), str(fname + '.wav'))
      display(Audio(str(fname + '.wav'), rate=16000))

      plt.figure(figsize=(14,5))
      ax=plt.axes(title=fname)
      ax.set_facecolor('black')

      plt.scatter(x,y, c=c)
      plt.xlabel("Time")
      plt.ylabel("Pitch")
      plt.show()

@asigalov61 Please upload this code to github, you can directly run the kind, thank you!
#@title Improv Generator
#@markdown Select desired instruments (any combination is fine)
Piano = True #@param {type:"boolean"}
Guitar = True #@param {type:"boolean"}
Bass = True #@param {type:"boolean"}
Violin = False #@param {type:"boolean"}
Cello = False #@param {type:"boolean"}
Harp = False #@param {type:"boolean"}
Trumpet = False #@param {type:"boolean"}
Clarinet = False #@param {type:"boolean"}
Flute = False #@param {type:"boolean"}
Drums = True #@param {type:"boolean"}
Choir = False #@param {type:"boolean"}
Organ = False #@param {type:"boolean"}
#@markdown Improv Timings and Velocity
desired_prime_time = 10 #@param {type:"slider", min:0, max:127, step:1}
desired_prime_duration = 12 #@param {type:"slider", min:1, max:126, step:1}
desired_velocity = 6 #@param {type:"slider", min:1, max:8, step:1}
#@markdown Model settings
number_of_tokens_tp_generate = 512 #@param {type:"slider", min:32, max:4064, step:32}
number_of_batches_to_generate = 4 #@param {type:"slider", min:1, max:16, step:1}
temperature = 1 #@param {type:"slider", min:0.1, max:1, step:0.1}
instruments = []
if Piano:
instruments += [
0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((0 * 128) + 60)+1152, # instrument number(1) + start pitch(60)
]
if Guitar:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((1 * 128) + 60)+1152, # instrument number(1) + start pitch(60)
]
if Bass:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((2 * 128) + 48)+1152, # instrument number(2) + start pitch(48)
]
if Violin:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((3 * 128) + 72)+1152, # instrument number(3) + start pitch(72)
]
if Cello:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((4 * 128) + 48)+1152, # instrument number(4) + start pitch(48)
]
if Harp:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((5 * 128) + 72)+1152, # instrument number(5) + start pitch(72)
]
if Trumpet:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((6 * 128) + 72)+1152, # instrument number(6) + start pitch(72)
]
if Clarinet:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((7 * 128) + 72)+1152, # instrument number(7) + start pitch(72)
]
if Flute:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((8 * 128) + 72)+1152, # instrument number(8) + start pitch(72)
]
if Drums:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((9 * 128) + 35)+1152, # instrument number(9) + start pitch(35)
]
if Choir:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((10 * 128) + 72)+1152, # instrument number(10) + start pitch(72)
]
if Organ:
instruments += [0, # delta start time (0)
((desired_prime_duration * 8) + (desired_velocity-1))+128,
((11 * 128) + 60)+1152, # instrument number(11) + start pitch(60)
]
instruments[0] = desired_prime_time
instruments[3::3] = [0] * len(instruments[3::3])
outy = instruments
#===================================================================
print('=' * 70)
print('Los Angeles Music Composer Model Improvisation Generator')
print('=' * 70)
print('Generation settings:')
print('=' * 70)
print('Model temperature:', temperature)
print('=' * 70)
print('Selected Improv sequence:')
print(outy)
print('=' * 70)
inp = [outy] * number_of_batches_to_generate
inp = torch.LongTensor(inp).cuda()
#start_time = time()
out = model.module.generate(inp,
number_of_tokens_tp_generate,
temperature=temperature,
return_prime=True,
min_stop_token=0,
verbose=True)
out0 = out.tolist()
print('=' * 70)
print('Done!')
print('=' * 70)
#print('Generation took', time() - start_time, "seconds")
print('=' * 70)
#======================================================================
print('Rendering results...')
print('=' * 70)
for i in range(number_of_batches_to_generate):
print('=' * 70)
print('Batch #', i)
print('=' * 70)
out1 = out0[i]
print('Sample INTs', out1[:12])
print('=' * 70)
if len(out) != 0:
  song = out1
  song_f = []
  tim = 0
  dur = 0
  vel = 0
  pitch = 0
  channel = 0

  son = []
  song1 = []

  for s in song:
    if s >= 128 and s < (12*128)+1152:
      son.append(s)
    else:
      if len(son) == 3:
        song1.append(son)
      son = []
      son.append(s)

  for ss in song1:

    tim += ss[0] * 10

    dur = (((ss[1]-128) // 8)+1) * 20
    vel = (((ss[1]-128) % 8)+1) * 15

    channel = (ss[2]-1152) // 128
    pitch = (ss[2]-1152) % 128

    song_f.append(['note', tim, dur, channel, pitch, vel ])

  detailed_stats = TMIDIX.Tegridy_SONG_to_MIDI_Converter(song_f,
                                                      output_signature = 'Los Angeles Music Composer',
                                                      output_file_name = '/content/Los-Angeles-Music-Composer-Music-Composition_'+str(i),
                                                      track_name='Project Los Angeles',
                                                      list_of_MIDI_patches=[0, 24, 32, 40, 42, 46, 56, 71, 73, 0, 53, 19, 0, 0, 0, 0],
                                                      number_of_ticks_per_quarter=500)

  print('=' * 70)
  print('Displaying resulting composition...')
  print('=' * 70)

  fname = '/content/Los-Angeles-Music-Composer-Music-Composition_'+str(i)

  x = []
  y =[]
  c = []

  colors = ['red', 'yellow', 'green', 'cyan', 'blue', 'pink', 'orange', 'purple', 'gray', 'white', 'gold', 'silver']

  for s in song_f:
    x.append(s[1] / 1000)
    y.append(s[4])
    c.append(colors[s[3]])

  FluidSynth("/usr/share/sounds/sf2/FluidR3_GM.sf2", 16000).midi_to_audio(str(fname + '.mid'), str(fname + '.wav'))
  display(Audio(str(fname + '.wav'), rate=16000))

  plt.figure(figsize=(14,5))
  ax=plt.axes(title=fname)
  ax.set_facecolor('black')

  plt.scatter(x,y, c=c)
  plt.xlabel("Time")
  plt.ylabel("Pitch")
  plt.show()

@asigalov61

I added your code, and when I ran it, I got an error that the model could not be found.I added your code, but I ran it and got an error, the model could not be found, I hope you will submit this code to github, my other code can run

@xuboot Sorry about your difficulties.
You need to load the model in the colab:
https://colab.research.google.com/github/asigalov61/Los-Angeles-Music-Composer/blob/main/Los_Angeles_Music_Composer.ipynb

@xuboot I will add it to the colab soon. Please give me some time.

@xuboot I added the code to the colab. Hope this will help you. :)


How do I set $revenue on revenue events with the SDK?

Both the LTV Guide and the Tracking Revenue guide say that you must set $revenue on revenue events in order to track LTV.
The revenue interface in the JavaScript SDK doesn't provide a way to set that, and when I inspect the raw events from the Amplitude browser extension, I don't see any $revenue property either. The event properties look like this:
"event_properties": {
    "productId": "Credits",
    "value": 2.99,
    "hasCoupon": false,
    "$productId": "Credits",
    "$quantity": 1,
    "$price": 2.99
  },

So how can I set $revenue? Or are the docs incorrect in saying I need to set that property?
Thanks,
Angus

Hello Angus @GusRuss89 ,
The help center has generic guides that do not rely on one individual language. Please take a look at the JavaScript guide for tracking revenue. Implementations in different languages vary wildly. Let us know if you have any more concerns about the Revenue API.
Dante

Hi @dantetam
Thanks for the response. I thought it might be the case that the JS SDK adds that property under the hood, but from what I can see in the raw event properties that get sent to Amplitude, there is no $revenue property.
So does the JS SDK do it correctly even though it doesn't add this property? Or is there no way to properly track LTV using the JS SDK? Or perhaps I'm supposed to set $revenue using Revenue.prototype.setEventProperties()?
Either way it seems that the two documentation pages I linked should be updated, because they both explicitly say that $revenue must be set on the event.
Thanks,
Angus

Hello Angus,
Yeah, you are right to notice there are discrepancies between API implementation across different languages, as well as discrepancies between the help center pages vs the developer docs. Regarding the documentation, they are meant to be two separate sets, for two different audiences. They are also maintained by two different teams. This repo and the engineering is monitored by us, the SDK engineering team. We'll try and align the documentation better.
Yes, to answer your question, the JS SDK should track LTV properly.

The preferred method of tracking revenue for a user is to use logRevenueV2() in conjunction with the provided Revenue interface. Revenue instances will store each revenue transaction and allow you to define several special revenue properties (such as 'revenueType', 'productIdentifier', etc.) that are used in Amplitude's Event Segmentation and Revenue LTV charts.

I assume you are looking for this LTV chart analytics feature on the Amplitude website.
CC: @jooohhn @ajhorst
Dante


Fix package position links

Splitting at the 44th character means leaving the source/ part which is not part of GitHub nixpkgs URLs. All "Nix expression" links in the package search are currently broken because of this.
This fix strips the first three path components instead (/nix/store/...-source).

Oops, I tried bumping the version in VERSION to trigger the import but now it 404s. I'm just gonna wait.

@ncfavier no need to bump the version, this is only needed when we change elasticsearch schema.

Noted.

@ncfavier Sorry I wrote to fast before. The VERSION change lets us test - alongside existing search index.

Ok, I assume it's something you have to do?

yes, sadly, It would be nice to automate this somehow.

I'll import the channels manually and then merge this.


PUB/SUB + pyobj needs documentation

I've been trying to transmit python objects using PUB/SUB and [send/recv]_pyobj. It turns out that this doesn't work out of the box, due to the way topics work with PUB/SUB. The subscriber needs to set a topic:
socket.setsockopt_string(zmq.SUBSCRIBE, "")

and the publisher needs to send the topic as the first part of the message
socket.send_string("", zmq.SNDMORE)
socket.send_pyobj(data)

The subscriber will have to eat the topic with an extra receive:
topic = socket.recv_string()
data = socket.recv_pyobj()

It would be nice if this were better documented - both how to send topics with send_pyobj, and that the topic is absolutely necessary in the first place (if the topic isn't set on the subscriber side, no messages go through).

The way PUB/SUB works, subscribers start in a completely unsubscribed state. By subscribing to the empty topic '', you are subscribing to all topics. If you do this, the additional topic frame of send(b'') is unnecessary, and you can use single send/recv_pyobj calls. You only need a separate topic frame if you are actually using topics, rather than having all subscribers subscribe to all topics. This works, for instance:
sub.subscribe(b'') # subscribe to all
pub.send_pyobj(data)
recvd_data = sub.recv_pyobj)

But as you've pointed out, if you have 'real' topics, the topic must be sent in a dedicated frame with send(topic, SNDMORE). Most of this ought to be covered in the zmq Guide.
Indeed, the send_<serialize> and recv_<deserialize> map exactly to single socket.send and socket.recv calls, so all of the send/recv/SNDMORE/RCVMORE semantics are exactly the same, including the relationship to the topic frame. The only addition is a single serialize/deserialize call, for convenience.


support proj4 6 for pyproj 2 compatibility

Checklist

[x] Used a fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Let's go with #58


Reimplement storage and summarization for performance

This PR changes the fundamental way routes are stored and the route summarization is calculated.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

251 of 269 (93.31%)  changed or added relevant lines in 4 files are covered.
6 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-4.2%) to 85.175%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

cmd/routesum/main.go
3
5
60.0%

pkg/routesum/rstrie/rstrie.go
136
142
95.77%

pkg/routesum/routesum.go
74
84
88.1%

Files with Coverage Reduction
New Missed Lines
%

pkg/routesum/routesum.go
6
83.84%

Totals

Change from base Build 622044836:
-4.2%

Covered Lines:
316

Relevant Lines:
371

💛  - Coveralls

Have you considered just using a specialized binary tree like mmdbwriter? I'd guess that it would be faster, use less memory, and it would allow you to output the networks in order.

Hi @oschwald! I did consider a specialized binary tree. @rafl told me about the existence of radix trees, and this PR implements a radix-ish tree of radix 2 in rstrie.go.  I haven't looked in detail at the structure implemented in mmdbwriter yet--I'll do that when things settle down a bit--but I'm pretty happy with the shape this one is taking. Does mmdbwriter do something better?

I saw the radix tree implementation. Nice!
The mmdbwriter data structure is less sophisticated but fairly well adapted to this particular problem. It eliminates the need to store the bits separately and makes merging networks very simple.

I created 3 datasets to use for speed comparisons:

All IPs in the GeoLite2 Country DB
All IPs in the GeoLite2 City DB
All IPs in the DShield API's Intelfeed endpoint

See mmdb-dump-networks and dshield-intelfeed-ips for utilities created for the purpose of generating these datasets.
I used go testing's built-in benchmarking functions to check the speed and memory performance of the old code vs the new code on these data sets.

Source
Num IPs
After Summing
Old Code (s)
Old Code (MB Allocated)
Old Code (Heap MB at Full Structure)
New Code (s)
New Code (MB Allocated)
New Code (Heap MB at Full Structure)

DShield API's Intelfeed IPs
58,131
41,765
63
6615.66
3.25
2.6
17.15
8.06

GeoLite2 Country Networks
1,843,336
96,325
More than 3 hours
DNF*
DNF*
3.1
812.64
29.59

GeoLite2 City Networks
15,530,913
96,325
DNT*
DNT*
DNT*
22.9
6637.09
27.59

* DNF: Ran for 3 hours and was killed by the testing platform. I could have overrode this even more, but I didn't see the need, as the new code processed the dataset in 3.1 seconds.
* DNT: Based on the performance of the old code on the previous dataset, and this being around 8 times larger, I didn't even try.
Based on the above, it's clear this PR greatly improves the speed and memory usage of the routesumming algorithm. #9 contains an illustration that makes performance of the original code look like it degrades exponentially as input increases, however the x-axis scale itself is essentially exponential so it's hard to determine whether it it actually exponential growth or just linear growth. I'll leave that analysis to another day.
The idea posed by @oschwald, i.e. to replace the radix-like trie with a binary tree, to not store any path-related information in the nodes themselves, but rather to derive it from the edges bewteen them, may yet prove to be an additional improvement. However based on the results above, the PR is good enough to merge currently.


RS display problem

hello, I have been using Sondehub for almost 3 years, but for about 8 days we have noticed a very big slowdown in the display of the RS and this on 3 PCs, 1 Tablet, and 1 Phone, it becomes very difficult to be able to follow a probe, and we are 5 people to have the same problem, the BETA version works perfectly well before, well done and thank you for your work

Thanks for the report @14ri004 . We are looking into options to resolve this, it seems to be regression from when we switched to websockets. Do you happen to know if this is a CPU, Memory or Network limitation?

hello, we've been doing it for about 8 days, it's lagging, very slow screen display, see blocked, the processor is running at maximum, with the BETA version everything was perfect, I use Intel Core 2 duo 2.0Ghz and 2.8Ghz, friends use I5 and I7 and the problem is the same, before the probe appears on the Sondehub screen, it happens 5 to see 10mn and whatever the internet connection and the PC used, I hope you find the solution, and thank you for your help.

Thanks for the information @14ri004 . as a work around, if you know the serial number for the radiosonde you can filter the website to just that serial, eg : https://sondehub.org/#!q=T0440081

The problem is that we cannot get the number, the probe is not visible on SondeHub, otherwise yes I know the manipulation, as long as there are not many websock messages, c  'is OK, but from 200/300 finished no longer possible to display the new probes, today after 1h30 and multiple restart of SondeHub, the probes that I wanted to follow were still invisible.

Hi @14ri004 I just pushed some changes to our testing branch that might help you out. Can you check https://testing.v2.sondehub.org when the next radiosonde is released and report back if this resolves your issue?

hello, yes ok we will do the tests and I will come back to you to inform you of the result, thank you

perfect, we tested the test probehub with 4 people? and the detection as well as the navigation in the page was very good, no more slowing down like yesterday, thank you

Thanks for the feedback :)


explorer supports modify ip-white-list

What type of PR is this?

[ ] bug
[ ] feature
[ ] enhancement

What problem(s) does this PR solve?
Issue(s) number:
Description:
How do you solve it?
Special notes for your reviewer, ex. impact of this fix, design document, etc:

https://github.com/vesoft-inc/nebula-explorer-ent/issues/1481


Check for array values in Parse.Query

Hello!
I have an array of colors and I have an user that also have an array of colors. I want to know if it's possible to check if user's array have at least one value of my array. If so, I want to get the user.
Both are array of strings.
Thanks!

@devfelipereis you can take a look here: https://parse.com/docs/js/guide#queries-queries-on-array-values and https://parse.com/docs/js/guide#queries-compound-queries


Can't see projects (but can open them)

No project is showing in open project dialog box, only overlapping All a and Browse workspace directory buttons (same like https://github.com/OpenRefine/OpenRefine/issues/3277).  I am able to open workspace directory and I am able to open the projects using direct link (http://<IP_ADDRESS>:3333/project?project=xxx).
To Reproduce
I have deleted one project directory from workspace directory.
Screenshots

Versions

Operating System: Windows 7
Browser Version: Chrome 89
OpenRefine: OpenRefine 3.4.1

Did you see this comment in the issue you referenced?
Can you try starting with the -d pointing to a directory with only ASCII characters to see if that helps?

@thadguidry - Do you know where the requirement for ASCII directory names comes from?

@gitonthescene not sure in the code.  It's a core Java lib.  @tfmorris would know.

@thadguidry You can reproduce the behavior described by @jkjkjkjkjk like this:
Steps to reproduce the problem

Fresh install of openrefine
Create first project in standard workspace directory and change some data
Create second project in standard workspace directory and change some data
Quit OpenRefine
Check the  "projectIDs"-array of workspace.json => it contains two valid project ids
Restart OpenRefine
Delete both projects  on the UI ("Open project")
Quit OpenRefine
Check the  "projectIDs"-array of workspace.json => it still contains the id of one of the deleted projects
Open OpenRefine => The screen shows only "create project" and "open project"
Create a new project and change some data
Quit OpenRefine
Reopen OpenRefine and the UI still only shows "create project" and "open project"
Quit OpenRefine and reopen it => the problem persists as the id of the deleted project remains in workspace.json

Solution
To resolve the issue I had to remove the invalid project id from the "projectIDs"-array of workspace.json
Tested with

macos 10.14.6 with OpenRefine 3.5 beta1 as well as OpenRefine 3.4.1 on Safari 14.1 and Firefox 88
windows 10 with OpenRefine 3.4.1 (incl Java8-JRE) on Edge.

The PATH to workspace directory contains only ascii characters.
Expected behaviour
Remove all ids of deleted projects from the "projectIDs"-array of workspace.json

Thanks, removing id of deleted project from workspace.json worked.

@WaltonG: I think you are right. So I will assign you on the basis you are working on I#1418.
Regards, Antoine


Adding LXRobotics logo to be shown as an adopter of UAVCAN

Is there a version with inverted color and transparent background though?

There is, but that's not in sync with LXRobotics CI (corporate identity).

Okay, whatever.


[Task] Update Chrome import instructions to include details about zipping the chrome folder

Is this really necessary after the extension is published to the browser stores?


Tutorial linked to on wiki is semantically out-of-date and only serves to confuse, should be removed.

As someone trying to understand how to use Celluloid and how it is useful I went to the tutorial "Introducing Celluloid: a concurrent object framework for Ruby". There is a warning that it is syntactically out of date. Unfortunately, it appears to also be semantically out of date. It states:
Since actors can only process one method call at a time, the call to Mapper#value will block until Mapper#run has completed, even though Mapper#run was called asynchronously.
Oh, I get this, I think. Each actors can only run one thing at a time, that's how they ensure thread-safety. They just changed the bang syntax to async. So I make the following program:
require 'celluloid'

class Sheen
  include Celluloid

  def initialize(name)
    @name = name
  end

  def set_status(status)
    puts "#{Time.now.to_f} start #{status}"
    sleep 1
    @status = status
    puts "#{Time.now.to_f} end #{status}"
  end

  def report
    "#{@name} is #{@status}"
  end
end

charlie = Sheen.spawn "Charlie Sheen"
charlie.async.set_status "asynchronously winning!"
charlie.async.set_status "asynchronously losing!"
puts Time.now.to_f
sleep 0.25
puts Time.now.to_f
puts charlie.report

To my surprise, neither of the statements above are true. Running charlie.report does not block, and even two async set_status methods are running at the same time. I wonder what the point of actors are then. I browse other pages on the wiki for answers but find none. I decide to give up on learning celluloid.
In this case, no documentation is probably better than wrong documentation.

You are right that the documentation is profoundly dated. A lot more than you said. The point of actors is plain once the correct behaviors of the current code is laid out. Now that we're at a release point, the documentation is ready to not only be updated but pieced apart to various gems various functionality was moved to. Sorry for the confusion, feel free to continue this as a Google Groups discussion, and I will post a ticket addressing documentation specifically.


Add specific Azure permissions required for Azure SD

There isn't any documentation that details which specific Azure permissions are required to use Azure SD. I found the proposed permissions to work through trial and error.

Thank you for your contribution.
Given that the Azure SD is still (and sadly) effectively unmaintained, I'm inclined to just merge this.
@roidelapluie @bboreham @machine424 as the general maintainers of this repo, WDYT?

Thanks for this. (Taking your word for it :))
Does this apply and make sense for all authentication methods (OAuth, ManagedIdentity, SDK)?

Yes