Taking 10 seconds on an empty repo,

I'm new to jest and I heard about it being slow so I was expecting it. But the main project I integrated Jest in was actually taking upwards of 2 mins! I might be doing something wrong, but even on this empty test repo it's taking 10+ seconds: https://github.com/laggingreflex/jest-test

Do you want to request a feature or report a bug?
Bug/question?
What is the current behavior?
Jest is taking upwards of 12 seconds on an empty repo - it only has 2 empty test files (empty describe and it with nothing in them), and no actual code or dependencies (other than jest).
If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal repository on GitHub that we can npm install and npm test.
https://github.com/laggingreflex/jest-test
What is the expected behavior?
it should be faster...
Run Jest again with --debug and provide the full configuration it prints. Please mention your node and npm version and operating system.
{
  "testRegex": "(/tests/.*|/src/.*\\.test)\\.js",
  "rootDir": "C:\\Test\\jest-test",
  "name": "C:-Test-jest-test",
  "setupFiles": [],
  "testRunner": "C:\\Test\\jest-test\\node_modules\\jest-jasmine2\\build\\index.js",
  "scriptPreprocessor": "C:\\Test\\jest-test\\node_modules\\babel-jest\\build\\index.js",
  "usesBabelJest": true,
  "automock": false,
  "bail": false,
  "browser": false,
  "cacheDirectory": "C:\\Users\\x\\AppData\\Local\\Temp\\jest",
  "clearMocks": false,
  "coveragePathIgnorePatterns": [
    "\\\\node_modules\\\\"
  ],
  "coverageReporters": [
    "json",
    "text",
    "lcov",
    "clover"
  ],
  "globals": {},
  "haste": {
    "providesModuleNodeModules": []
  },
  "mocksPattern": "__mocks__",
  "moduleDirectories": [
    "node_modules"
  ],
  "moduleFileExtensions": [
    "js",
    "json",
    "jsx",
    "node"
  ],
  "moduleNameMapper": {},
  "modulePathIgnorePatterns": [],
  "noStackTrace": false,
  "notify": false,
  "preset": null,
  "preprocessorIgnorePatterns": [
    "\\\\node_modules\\\\"
  ],
  "resetModules": false,
  "testEnvironment": "jest-environment-jsdom",
  "testPathDirs": [
    "C:\\Test\\jest-test"
  ],
  "testPathIgnorePatterns": [
    "\\\\node_modules\\\\"
  ],
  "testURL": "about:blank",
  "timers": "real",
  "useStderr": false,
  "verbose": null,
  "watch": false,
  "cache": true,
  "watchman": true,
  "testcheckOptions": {
    "times": 100,
    "maxSize": 200
}

node -v
v6.9.0
npm -v
3.10.8
ver
Microsoft Windows [Version 10.0.14946]

Is it possible your anti-virus software is invoked on every file system access within your project? Can you try whitelisting this folder and re-running Jest? This is a common problem on Windows.

I [disabled anti-virus completely][1], also disabled firewall and search indexing just to be sure and rebooted but still no improvement.
I tried a couple different options, Docker, Windows Ubuntu bash, and it does perform better in those (in some cases) down to 6 secs on this test project and on my original project from 2mins to 20 secs. But those systems seem to have problems of their own making them not ideal for development.
So this is definitely a Windows issue, but not related to anti-virus. Please let me know if and how I can help debug this further. I would really love to use Jest but natively on Windows!

Comparison on other systems I tried and their shortcomings:

OS/System
Time

Windows (native)
12s

Docker with Node image and mounted volume (docker -v"%CD%:/test")
10s

Docker with Node image and copying the contents (COPY . .)
6s

Windows Ubuntu Bash (under /mnt/c)
20s

Windows Ubuntu Bash (VolFs /home/)
6s

Windows Ubuntu Bash works best but with VolFs which isn't ideal for development because files in that system don't work well with native Windows tools (editing outside of bash breaks them).
Docker works well but only when you copy the contents to the image, which won't work for active development. Unfortunately mounted volumes has it's own issues with Docker (on Windows at least) which causes it to be still slow. This is particularly visible on a bigger project.
Windows Ubuntu Bash with mounted filesystem (DriveFS /mnt/c/...), which is the ideal file system for active development, it's actually worse, taking 20 secs!
I saw improvement on my original project which was taking 2+ mins which has come down to ~20 secs with Docker and Windows Ubuntu Bash with native VolFs, but neither of those options seem ideal.

On Windows we cannot use either watchman or the find binary so we fall back to using node.js to crawl the repo: https://github.com/facebook/jest/blob/master/packages/jest-haste-map/src/crawlers/node.js#L23 – if you can send a PR to make any of this faster, that would be awesome!
We have to do this on every startup of Jest, unfortunately, to do static analysis (like for the watch mode or for jest -o etc.).

I found one problem, it monitors .git folder by default. And it seems this happens on every file change too:

(putting console.log(directory) here))
Specifying "testPathDirs" resolves this, but it should ignore .git by default.
Although as you see, this by itself was still pretty quick. The delay almost certainly seems to be on the "RUNS". Could you point to the file where "RUNS" happen in the code?

When Jest prints "RUNS" it means it is already running the code inside your tests. Maybe you can add some instrumentation (console.time(), console.timeEnd()) there?

So I did
console.time('startSync');
console.time('startASync');
describe('simple', () => {
  it('test', () => {
    console.timeEnd('startASync');
  });
});
console.timeEnd('startSync');

 PASS  tests\test.js
  simple
    √ test (9ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        11.381s
Ran all test suites related to changed files.

Watch Usage
 › Press a to run all tests.
 › Press o to only run tests related to changed files.
 › Press p to filter by a filename regex pattern.
 › Press q to quit watch mode.
 › Press Enter to trigger a test run.
  console.log console.js:77
    startSync: 7.539ms

  console.log console.js:77
    startASync: 32.101ms

As seen, the time it takes to run (reported by console.time and by Jest itself (√ test (9ms))) was expectedly within under a second.
But the RUNS still displays for 11 secs before it shows PASS. I found that here is where it prints RUNS. Now since console.log has an issue* so I just changed it to display Date.now. And did the same with PASS (which was here), and the difference was the same that it showed in Time:        11.381s
It's definitely doing something other than just running the test files between the time it displays RUNS and PASS
And since *console logs are globbed up by Jest and only shown at the end (bw any way to turn this off temporarily?) I can't tell whether it starts running the file as soon as it displays "RUNS" or after 11 secs, mainly in the last few seconds. I'm betting on the latter.
PS: Also notice the timers startASync = 32ms and startSync = 7.5ms, and the √ test (9ms), shouldn't it reflect the startASync time (the time it takes to get to the end an it)?

It just looked at it, the test run should certainly complete in less than a second. This is really odd.
Is it at all different if you do jest -i? If not, can you try to profile things earlier? Check out https://github.com/cpojer/jest/blob/master/packages/jest-cli/src/TestRunner.js#L1

jest -i didn't help, unfortunately.
I did a node --prof, here's the processed file: processed2.txt
I've narrowed it down to the following statements that are causing the delay:
jest-cli/src/TestRunner.js#L304
// till here - 0.1s
return runTest(…)
  .then(()=>{
    // after here: 8s

jest-cli/src/runTest.js#L50
// till here - 0.5s
return TestRunner(…)
  .then(()=>{
    // after: 8s

jest-jasmine2/src/index.js#L110
// till here - 2s
runtime.requireModule(testPath);
// after: 7s

After this it goes into jest-runtime's requireModule which _execModule's a lot of modules (is that expected behaviour? even with autoMock:false?), so timing it didn't make sense, at least for the test file because lots of other module files are also collectively being loaded and might be what's causing the delay. Thoughts on this?

Interesting. Are you using npm2? I would recommend to upgrade to yarn or npm3 to install Jest which will result in a lot fewer modules that will be required per context.

Closing for now as I don't think this is actionable. If there are issues in Jest that we can fix, please comment here or send a PR and I'll reopen this issue.

Sorry I couldn't debug this further earlier.
I was actually using npm3 and yarn. In fact in all these tests I had no other dependencies than Jest itself (which was set up via Lerna).
You didn't answer my question earlier: [...] After this it goes into jest-runtime's requireModule which _execModule's a *lot of modules* -- is that expected behaviour? even with autoMock:false? Shouldn't these modules only be loaded when autoMock:true?

Jest does require a ton of modules, yes, but normally that shouldn't take more than ~10ms. The paths very much look like the node_modules folder wasn't deduplicated, which is really odd. You should only have jest-* modules on the top level of your node_modules folder. Would you mind wiping away node_modules and installing Jest again?

I'm having the same problems on node 6.9.3 and npm 3.10.10. Only testing one simple file:
// src/lib/tests/jest-test.spec.js
describe('wrapRequest', () => {
  it('should pass this test', () => {
    expect(1).toBe(1);
  });
});

jest "src/lib/tests/jest-test.spec.js"

 PASS  src\lib\tests\jest-test.spec.js (21.568s)
  wrapRequest
    √ should pass this test (9ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        45.878s
Ran all test suites matching "src/lib/tests/jest-test.spec.js".

46 seconds is outrageous. I have already tried nuking node_modules but no cigar. I am not using babel-jest. On Windows 10 x64 and using Jest 18.

Consequent runs take ~7 seconds:
jest "src/lib/tests/jest-test.spec.js"

 PASS  src\lib\tests\jest-test.spec.js
  wrapRequest
    √ should pass this test (9ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        7.05s, estimated 22s
Ran all test suites matching "src/lib/tests/jest-test.spec.js".

Same for me, I'm running enzyme example repo and it takes 7-10 seconds. The assertions itself takes only 50ms.
I'm running Windows 7, node 6.9.2, npm 3.10.9.
I will try to track it down - any suggestions?

@apieceofbart You could start by confirming whether you're getting the same delays as me on breakpoints I mentioned above https://github.com/facebook/jest/issues/1986#issuecomment-256936218
Specially the last one:

jest-jasmine2/src/index.js#L110
// till here - 2s
runtime.requireModule(testPath);
// after: 7s

runtime.requireModule(testPath) here for me took 5s! The problem most likely lies somewhere around here.

@laggingreflex  thanks!
It's a bit different in my case.
This line, takes over 3s: https://github.com/facebook/jest/blob/master/packages/jest-cli/src/runTest.js#L32
From there till the end it's similar to your case. I will keep posting other findings.

@laggingreflex
I can pretty much confirm your comment: https://github.com/facebook/jest/issues/1986#issuecomment-256936218
Most of the time is spent running _execModule for every module in node_modules (over 2170 files in enzyme example repo).
This is not the only performance issue but certainly the main problem.

@apieceofbart can you try testEnvironment: "node" in the config? The 2s overhead is odd; I'm assuming it is because of loading jsdom.
Jest has to read files from disk to load and run them. What you are pointing to are the pieces of code that executes node modules and user code. It may be that your harddrive is very slow or your anti-virus scanner looks at all the node_modules. If you make an exception for your projects, all your tools may be faster.

@cpojer

@apieceofbart can you try testEnvironment: "node" in the config? The 2s overhead is odd; I'm assuming it is because of loading jsdom.

bingo! This brings creating test env from 2s to 15ms. However this runtime.reqiureModule still takes around 2-3s. Also 'jest-runtime' constructor takes almost a second.
I have just bought a new SSD laptop and whitelisted my projects folder in anti-virus scanner (even removing the scan completely) and it doesn't help. Thanks for you concern, I will let you know if find anyhing.

I had similar observation, running jest inside Docker on macOS with alpine-node image (and the app bootstrapped with create-react-app).
I was running yarn test -- --coverage to capture the time. I run the tests 5 times each, and below I present the average results.
Results

Content location
Time (tests)
Done in (all)

content inside the container (COPY . /app)
~2.1s
~3.4s

content mounted from the host (-v $(pwd):/app)
~9.6s
~31.5s

Reason
In my case, the cause was the well known problem of slow Docker mounts on macOS. The whole issue thread here: https://github.com/docker/for-mac/issues/77
Solution (until the mounts get faster)
For now, I just mount my ./src (-v $(pwd)/src:/app/src) directory, instead of the whole ./app (so local ./node_modules and ./package.json from inside the container are used). This allows tests to run fast, but adds a bit of extra work, when you need to install additional node package (you need to rebuild the image).

For Windows, could this and various similar issues on yarn be related?: https://github.com/yarnpkg/yarn/issues/1496 https://github.com/yarnpkg/yarn/issues/990 (just a hunch, haven't checked these out in detail). In https://github.com/yarnpkg/yarn/pull/2958 OP mentioned a Windows specific nodejs bug in fs https://github.com/nodejs/node/issues/2069 that was only recently fixed (v7.1).

We have a similar problem, we are importing stylefmt and it's taking around 4-5s to get resolved using console.time as below
console.time("stylefmt spec");
const style = require("stylefmt");
console.timeEnd("stylefmt spec");

It seems this happens when its using jest-resolve, when I used jasmine/node normally on the same exact project its taking around 1-2s
Not entirely sure if its related or if jest have a threshold but after loading this specific test its failing other tests without a message (however the tests alone works). This is the error im getting
 FAIL  src\utils\string.spec.ts
  ● Test suite failed to run

    Error: No message was provided

I am using Jest (and enzyme) in create-react-app and adding a --runInBand flag brought me from around 20 sec (only 8 tests) down to 3.3 sec. on a new Windows 10 maschine. Might be useful in other cases as well. I still feel that something is off with my setup but at least its not as bad.


Elasticsearch giving incomplete results when performing keyword searches

Try api.keyword_search('Boston').print_tables() or
hits = self._store_client.search_keywords(
    keywords='Boston', elasticfieldname=kw_source, max_hits=max_results)
drs = DRS([x for x in hits], Operation(OP.KW_LOOKUP, params=[kw]))

In both cases, the results they give back is obviously incomplete.

Closed because max results was the culprit here.


moveTo return promise/obs to indicate done

i have often in my game the need to perform either an emote or changing the facing direction after a finished move. E.g I can order an NPC to start gathering resources at XY and once he arrives there he/she confirms the action.
i have a helper like this
private async waitForCurrentMovementStopped() {
    return firstValueFrom(this.scene
      .gridEngine
      .movementStopped()
      .pipe(
        filter(m => m.charId === this.id),
        take(1)
      ));
  }

and wondered whether moveTo could return either a promise (preferably) or obs so that can be handled in one go?

moveTo() already returns an observable: https://annoraaq.github.io/grid-engine/api/methods.html#moveto
It should do just what you need.

hmm I could swear I checked the return type of the function and it said void. Will double check it later. thx for the hint

I'm sorry it seems my vscode project had some troubles. after a wipe of npm_modules and re-installing I can see the intellisense suggestions now properly and the returned obs.

hey @Annoraaq, I've just realized it's not exactly the same as my waitForCurrentMovementStopped example above. The observable fires way too soon even though the movement is not yet finished.

Hey @zewa666.
You are right, the observable emits when the movement starts. It is like positionChangeStarted() in that sense.
I am not sure whether it would be a good idea to change that because in case you need it to emit on positionChangeStarted, you could not get that anymore.
Or to put it another way: it is easy to change the observable to wait for movementStopped. But if it emits when the movement has finished, you can't make it fire earlier anymore, if you know what I mean. So I think it is more flexible the way it is right now.
But you should be able to easily modify the observable to do what you want:
this.gridEngine.moveTo('player', ...).pipe(
  switchMap(() => this.gridEngine.movementStopped().pipe(
      rxjs.filter(({ charId }) => charId == 'player')
  ))).subscribe(...);

exaclty what I wanted to propose with the config override. The description of the method gave me the wrong impression

. It will return an observable that will fire whenever the moveTo movement is finished or aborted

so it sounded somewhat as after done. but to male things easier, keeping the current behavior and adding a config option would be definitely helpful as it saves a couple of operator imports and LOC

I created an issue for that #241


Upgrade sqlx to 0.7

Migrated from #21 to refresh the pipeline.

f13a5d5 chore!: upgrade sqlx to 0.7
This required adding the rt-multi-thread feature to tokio in dev,
since sqlx no longer enables it.
BREAKING CHANGE: The library does not support MSSQL anymore, as SQLx
dropped support

@cking apoligies for the delay – this is now released! https://crates.io/crates/axum-sqlx-tx/0.6.0/dependencies


fix: added optional chaining in LayoutTopNavigation.vue (NTR)

Thank you @niklaswolf!
Merged with https://github.com/shopware/frontends/commit/ee601feb0813b8b5a117cf9eaf0fad1b4d142418
sorry, we have an issue with gitlab mirroring so it's not merged via GH, we'll fix that soon but you'll be mentioned in the version changelog!


model detects hand as a face. I cant retrieve confidence score from  tfjs-models/face-detection

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
last version
Custom code
Yes
OS platform and distribution
No response
Mobile device
No response
Python version
No response
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
In mediapipe model documentation (see: https://drive.google.com/file/d/1d4-xJP9PVzOvMBDgIjz6NhvpnlG9_i0S/preview) it says that alongside with bounding box coordinates and keypoints, confidence score should be also returned.
But there is no info about that in face-detection github repo and in the code.
In face-detection Demo, you can see for yourself that fists are displayed as faces. I need confidence score to prevent this action.
https://storage.googleapis.com/tfjs-models/demos/face-detection/index.html?model=mediapipe_face_detector
Standalone code to reproduce the issue
https://storage.googleapis.com/tfjs-models/demos/face-detection/index.html?model=mediapipe_face_detector

Try to show different hand movements, its being detected as face (mostly fist)

Relevant log output
No response

@DiannaShonia Please ensure that you're using a model version that provides confidence scores. Some older versions might not provide the same. Also try to raise the minimum confidence score required for a detection to be considered a face. This can filter out less confident detections, potentially reducing false positives.
This issue is related to tfjs so for any further questions , could you please post this issue in tfjs repository?
Thank you!


[chore]: add @dima74 to the CODEOWNERS

going to approve after addition to hyperledger organization

We can proceed with merging this.


Links inside docs page after staticpkg are broken

Describe the bug
After running staticpkg, attempt to navigate inside docs page fails (404 - page not found).
To Reproduce
Steps to reproduce the behavior:

Run pxt staticpkg and http-server -c-1 built/packaged in custom pxt target
Go to http://localhost:8080/docs/blocks/loops.html
Click on any of present blocks.
See 404 error - page not found.

Expected behavior
Docs about given block should be displayed properly.
Desktop (please complete the following information):

OS: Windows 10
Browser: Google Chrome
Version: 69.0.3497.100

Additional context

pxt-core version: 4.0.5
When directly opening http://localhost:8080/docs/blocks/loops/for.html, page is displayed correctly.
To my understanding, only links associated to blocks are broken, because when I click on for link in See also section, page is displayed properly. Navigating through left=hand side menu also seems to work fine.

May be related to https://github.com/Microsoft/pxt/issues/4700#issuecomment-423470461

This should have been fixed by https://github.com/Microsoft/pxt/pull/4704

There is definitely an improvement, but for example clicking on for block in loops category is still causing a 404 error.


Cmake error in Android Studio : find: ‘/cmake’: No such file or directory

When I tried to use the FreeRDP of Android,I follow the docs and run the below mentioned command in the terminal:
./scripts/android-build-freerdp.sh --ndk /home/Android/Sdk/ndk/25.0.8775105 --sdk /home/Android/Sdk
And getting the below mentioned error:
find: ‘/cmake’: No such file or directory
And camke is already installed in my system. When i checked the version of the cmake in the terminal . i am getting the below mentioned installed version :
cmake --version
cmake version 3.23.3
Can you please let me know the reason of the above issue.?
So i can proceed further my development.
Thanks.

you are lacking the CMake in android SDK


Menu drop down on usage reports section missing documentation for call diagnostics reporting

In Teams Admin Center under Usage Reports there is a Call Diagnostics report that is not represented in our documentation.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 78ade1a4-092c-1132-44df-ee2c683fdd33
Version Independent ID: 396b1aee-2f40-9895-9b00-07e38d11d31b
Content: Microsoft Teams analytics and reporting - Microsoft Teams
Content Source: Teams/teams-analytics-and-reports/teams-reporting-reference.md
Service: msteams
GitHub Login: @cichur
Microsoft Alias: v-cichur

Hi @Erikvg24
I'll look into this issue. Cindy

Hi @Erikvg24
I'll look into this issue. Cindy

Hi,
From our expert:
No, Scheduled report functionality is not there yet for our usage reports.
However – Customers can get few of the critical reports (Like user activity report ) through graph API. We are also working with MAC team to get additional graph APIs for other reports.
Hope this helps.

@Erikvg24  Thank you for submitting feedback. We understand that this issue has been resolved.
Please feel free to re-open this issue if there is a specific area of the docs that we can improve or make better. Thank you.


[CARBONDATA-3652] Make insert and select table columns equal

Why is this PR needed?
Described in https://issues.apache.org/jira/browse/CARBONDATA-3652
What changes were proposed in this PR?
Insert and select table columns should equal.
Does this PR introduce any user interface change?

No

Is any new testcase added?

Yes

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.2/1433/

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/1445/

retest this please

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/1482/

LGTM

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/1487/

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/1489/

LGTM

PR changes are already merged to master. Closing the PR

@shenh062326 you can close this PR, changes are already present in master.

@shenh062326 you can close this PR, changes are already present in master.

ok, I will close it.


Enable Redis tests in cluster mode

Add support to enable run k6 tests against Redis in cluster mode.
Add docker-compose to create a test redis cluster.
Add new k6 example.
Update README.

Hi @imiric,
Sorry for the delay in responding to this topic. I have come back to it and I think the easiest way to connect to Redis in this scenario, is using the new UniversalClient. I close this PR in favor of the PR: https://github.com/grafana/xk6-redis/pull/2.
If it's ok with you, we can discuss any question in the new PR.
Thank you very much!


Benchmark bytes / bytebuffer, protobuf vs zipkin vs wire.

For #2435
This is just a hacky set of code to compare performance of decoding ByteBuffer and byte[]. I used the protobuf library mainly since it has native decoders for each type so is easiest to compare. Naturally zipkin's Buffer could be updated to act on ByteBuffer directly to avoid this, though I'm not sure it'd be faster since protobuf uses Unsafe when it can. wire does especially poorly with ByteBuffer it seems, since it just copies it into a byte[] but doesn't have netty's unsafe optimizations like ByteBufUtil.getBytes would have.
As expected, wire is especially fast in time here, probably because it doesn't do hex decoding - it's nowhere near an apple comparison and mainly for reference. Otherwise, byte[] seems to be a bit faster, which isn't unexpected given the JVM is optimized for it, but the difference is mainly in the noise throughout. This could possibly be considered enough reason to switch to ByteBuffer though as given similar parsing time, ByteBuffer version is a bit better since it generates no garbage and could potentially reduce GC.
 I think this is still important eventhough the parsing side is less so (it does help in wonky huge dep clash environments like spark)
in our server, we have some flexibility, and to that end I was thinking about having the ability to override the codecs towards your initial issue #2435. When I started optimizing the Elasticsearch side, I noticed that our v2 format is so easy now, even converting from a codec-specific type like proto.span is still cheaper than us doing it directly for some reasons you mentioned.  While using google's proto library would re-introduce their version thrash I do like doing comparisons. I've thought most about wire as that is so incredibly more efficient than ours, and it has no notable deps we don't already depend on.
Thanks for doing this comparison!

Yeah it'd definitely only be for the server. And while protobuf-java-util has some dependencies, just protobuf-java is dependency free and I believe has never broken backwards compatibility so I wouldn't write it off so fast ;)
I added a wire-based decoder too, with the same code flow as the protobuf-based one, just uses wire for the decoding. Now that it is doing hex decoding, it's a proper comparison. Wire is quite a bit slower, probably since it has to do an array copy before converting to hex since it doesn't expose the ability to read single bytes, and maybe because protobuf uses Unsafe though I don't know if that really makes a big impact on modern JVMs (it would have a larger impact with java 8 than java 11 I guess).
Protobuf vs zipkin seems to show protobuf a few percent faster.
Garbage-wise, as expected the lack of an initial array copy does remove a whole buffer's worth of allocation, but due to the lots of allocations to create hex strings, it's a very small percentage overall, ~3%, so just a little improvement. This is after applying a change to reuse char[], which had a similar improvement in the base case too.
There do seem to be some differences shown here, though they're not so huge as to make it clear whether any approach is the best.
Benchmark                                                                           (num)    Mode   Cnt         Score         Error   Units
ProtoCodecBenchmarks.bytes_copy_protobufDecoder                                     10000  sample  1309     11510.855 ±     138.341   us/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:bytes_copy_protobufDecoder·p0.00    10000  sample           10010.624                 us/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:bytes_copy_protobufDecoder·p0.50    10000  sample           10977.280                 us/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:bytes_copy_protobufDecoder·p0.90    10000  sample           13156.352                 us/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:bytes_copy_protobufDecoder·p0.95    10000  sample           15319.040                 us/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:bytes_copy_protobufDecoder·p0.99    10000  sample           16967.270                 us/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:bytes_copy_protobufDecoder·p0.999   10000  sample           24244.060                 us/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:bytes_copy_protobufDecoder·p0.9999  10000  sample           25493.504                 us/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:bytes_copy_protobufDecoder·p1.00    10000  sample           25493.504                 us/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.alloc.rate                      10000  sample    15      1212.778 ±      22.951  MB/sec
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.alloc.rate.norm                 10000  sample    15  21949689.300 ±     213.505    B/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.churn.G1_Eden_Space             10000  sample    15      1110.207 ±     113.046  MB/sec
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.churn.G1_Eden_Space.norm        10000  sample    15  20094766.898 ± 2030230.821    B/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.churn.G1_Old_Gen                10000  sample    15       112.514 ±      11.349  MB/sec
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.churn.G1_Old_Gen.norm           10000  sample    15   2036516.677 ±  203864.604    B/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.churn.G1_Survivor_Space         10000  sample    15         0.443 ±       0.438  MB/sec
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.churn.G1_Survivor_Space.norm    10000  sample    15      8009.395 ±    7904.907    B/op
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.count                           10000  sample    15        82.000                counts
ProtoCodecBenchmarks.bytes_copy_protobufDecoder:·gc.time                            10000  sample    15       346.000                    ms
ProtoCodecBenchmarks.bytes_copy_wireDecoder                                         10000  sample  1068     14170.764 ±     135.522   us/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:bytes_copy_wireDecoder·p0.00            10000  sample           12648.448                 us/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:bytes_copy_wireDecoder·p0.50            10000  sample           13713.408                 us/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:bytes_copy_wireDecoder·p0.90            10000  sample           15810.560                 us/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:bytes_copy_wireDecoder·p0.95            10000  sample           16846.029                 us/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:bytes_copy_wireDecoder·p0.99            10000  sample           19930.808                 us/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:bytes_copy_wireDecoder·p0.999           10000  sample           24186.126                 us/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:bytes_copy_wireDecoder·p0.9999          10000  sample           24281.088                 us/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:bytes_copy_wireDecoder·p1.00            10000  sample           24281.088                 us/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.alloc.rate                          10000  sample    15      1129.202 ±      28.012  MB/sec
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.alloc.rate.norm                     10000  sample    15  25120909.386 ±     202.273    B/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.churn.G1_Eden_Space                 10000  sample    15      1040.511 ±      54.956  MB/sec
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.churn.G1_Eden_Space.norm            10000  sample    15  23153464.194 ± 1194946.337    B/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.churn.G1_Old_Gen                    10000  sample    15        90.519 ±       4.650  MB/sec
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.churn.G1_Old_Gen.norm               10000  sample    15   2014291.467 ±  102517.266    B/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.churn.G1_Survivor_Space             10000  sample    15         0.838 ±       0.323  MB/sec
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.churn.G1_Survivor_Space.norm        10000  sample    15     18578.527 ±    6846.810    B/op
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.count                               10000  sample    15        76.000                counts
ProtoCodecBenchmarks.bytes_copy_wireDecoder:·gc.time                                10000  sample    15       173.000                    ms
ProtoCodecBenchmarks.bytes_protobufDecoder                                          10000  sample  1340     11252.910 ±     126.595   us/op
ProtoCodecBenchmarks.bytes_protobufDecoder:bytes_protobufDecoder·p0.00              10000  sample            9912.320                 us/op
ProtoCodecBenchmarks.bytes_protobufDecoder:bytes_protobufDecoder·p0.50              10000  sample           10731.520                 us/op
ProtoCodecBenchmarks.bytes_protobufDecoder:bytes_protobufDecoder·p0.90              10000  sample           12546.867                 us/op
ProtoCodecBenchmarks.bytes_protobufDecoder:bytes_protobufDecoder·p0.95              10000  sample           14974.976                 us/op
ProtoCodecBenchmarks.bytes_protobufDecoder:bytes_protobufDecoder·p0.99              10000  sample           16912.548                 us/op
ProtoCodecBenchmarks.bytes_protobufDecoder:bytes_protobufDecoder·p0.999             10000  sample           19200.541                 us/op
ProtoCodecBenchmarks.bytes_protobufDecoder:bytes_protobufDecoder·p0.9999            10000  sample           19267.584                 us/op
ProtoCodecBenchmarks.bytes_protobufDecoder:bytes_protobufDecoder·p1.00              10000  sample           19267.584                 us/op
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.alloc.rate                           10000  sample    15      1130.991 ±      25.879  MB/sec
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.alloc.rate.norm                      10000  sample    15  20009660.550 ±     259.757    B/op
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.churn.G1_Eden_Space                  10000  sample    15      1136.615 ±     111.000  MB/sec
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.churn.G1_Eden_Space.norm             10000  sample    15  20105313.287 ± 1867925.592    B/op
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.churn.G1_Old_Gen                     10000  sample    15         0.319 ±       0.197  MB/sec
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.churn.G1_Old_Gen.norm                10000  sample    15      5659.194 ±    3507.786    B/op
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.churn.G1_Survivor_Space              10000  sample    15         0.354 ±       0.454  MB/sec
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.churn.G1_Survivor_Space.norm         10000  sample    15      6261.311 ±    8110.558    B/op
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.count                                10000  sample    15        84.000                counts
ProtoCodecBenchmarks.bytes_protobufDecoder:·gc.time                                 10000  sample    15       352.000                    ms
ProtoCodecBenchmarks.bytes_wireDecoder                                              10000  sample  1027     14696.033 ±     274.890   us/op
ProtoCodecBenchmarks.bytes_wireDecoder:bytes_wireDecoder·p0.00                      10000  sample           12697.600                 us/op
ProtoCodecBenchmarks.bytes_wireDecoder:bytes_wireDecoder·p0.50                      10000  sample           13991.936                 us/op
ProtoCodecBenchmarks.bytes_wireDecoder:bytes_wireDecoder·p0.90                      10000  sample           16420.045                 us/op
ProtoCodecBenchmarks.bytes_wireDecoder:bytes_wireDecoder·p0.95                      10000  sample           17812.685                 us/op
ProtoCodecBenchmarks.bytes_wireDecoder:bytes_wireDecoder·p0.99                      10000  sample           26746.552                 us/op
ProtoCodecBenchmarks.bytes_wireDecoder:bytes_wireDecoder·p0.999                     10000  sample           63427.052                 us/op
ProtoCodecBenchmarks.bytes_wireDecoder:bytes_wireDecoder·p0.9999                    10000  sample           64225.280                 us/op
ProtoCodecBenchmarks.bytes_wireDecoder:bytes_wireDecoder·p1.00                      10000  sample           64225.280                 us/op
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.alloc.rate                               10000  sample    15      1003.950 ±      33.760  MB/sec
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.alloc.rate.norm                          10000  sample    15  23180993.238 ±     327.812    B/op
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.churn.G1_Eden_Space                      10000  sample    15      1001.456 ±     113.305  MB/sec
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.churn.G1_Eden_Space.norm                 10000  sample    15  23105838.211 ± 2265066.780    B/op
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.churn.G1_Old_Gen                         10000  sample    15         0.585 ±       0.150  MB/sec
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.churn.G1_Old_Gen.norm                    10000  sample    15     13499.422 ±    3403.060    B/op
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.churn.G1_Survivor_Space                  10000  sample    15         0.752 ±       0.249  MB/sec
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.churn.G1_Survivor_Space.norm             10000  sample    15     17272.859 ±    5177.973    B/op
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.count                                    10000  sample    15        73.000                counts
ProtoCodecBenchmarks.bytes_wireDecoder:·gc.time                                     10000  sample    15       158.000                    ms
ProtoCodecBenchmarks.bytes_zipkinDecoder                                            10000  sample  1220     12361.137 ±     243.588   us/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:bytes_zipkinDecoder·p0.00                  10000  sample            9863.168                 us/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:bytes_zipkinDecoder·p0.50                  10000  sample           11452.416                 us/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:bytes_zipkinDecoder·p0.90                  10000  sample           15382.938                 us/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:bytes_zipkinDecoder·p0.95                  10000  sample           18246.861                 us/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:bytes_zipkinDecoder·p0.99                  10000  sample           22308.127                 us/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:bytes_zipkinDecoder·p0.999                 10000  sample           26914.292                 us/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:bytes_zipkinDecoder·p0.9999                10000  sample           27066.368                 us/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:bytes_zipkinDecoder·p1.00                  10000  sample           27066.368                 us/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.alloc.rate                             10000  sample    15       984.706 ±      72.183  MB/sec
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.alloc.rate.norm                        10000  sample    15  19129801.132 ±     226.638    B/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.churn.G1_Eden_Space                    10000  sample    15       993.104 ±     126.048  MB/sec
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.churn.G1_Eden_Space.norm               10000  sample    15  19270497.313 ± 1683336.845    B/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.churn.G1_Old_Gen                       10000  sample    15         0.434 ±       0.268  MB/sec
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.churn.G1_Old_Gen.norm                  10000  sample    15      8434.596 ±    5290.023    B/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.churn.G1_Survivor_Space                10000  sample    15         0.753 ±       0.651  MB/sec
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.churn.G1_Survivor_Space.norm           10000  sample    15     14673.383 ±   12535.509    B/op
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.count                                  10000  sample    15        83.000                counts
ProtoCodecBenchmarks.bytes_zipkinDecoder:·gc.time                                   10000  sample    15       335.000                    ms

rebased your branch as there were some merge conflicts due to drifts

I'm playing around a bit about the hex it is yeah double array allocation indeed

one thing I was thinking about in this change is that when you are ultimately backed by an array, google sneakily looks to see if that's the case and special cases accordingly.
When we have an array is unlikely something we'd optimize for in the grpc listener. I'm wondering if we can get a more realistic source like what armeria would be using and apply that to the bytebuffer (or otherwise) case...
PS I also looked at wire and started to play with inlining a hex utility.

cc also @swankjesse as this sort of thing is fun..
I think special peeking aside, it might boil down to if we can manage to read proto bytes as hex. if we can definitely wire should be better than it is now..
I added this to wire's ProtoReader though haven't tested it quite yet
  // stolen from okio
  private val HEX_DIGITS =
      charArrayOf('0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f')

  @Suppress("NOTHING_TO_INLINE")
  internal inline fun hex(data: BufferedSource, byteCount: Int): String {
    val result = CharArray(byteCount * 2)
    var i = 0
    while (i < result.size) {
      val b = data.readByte();
      result[i++] = HEX_DIGITS[b shr 4 and 0xf]
      result[i++] = HEX_DIGITS[b       and 0xf] // ktlint-disable no-multi-spaces
    }
    return String(result)
  }

  /**
   * Reads a `bytes` field value from the stream as a lower-hex string. The length is read from the
   * stream prior to the actual data.
   */
  @Throws(IOException::class)
  fun readBytesAsHex(): String {
    val byteCount = beforeLengthDelimitedScalar()
    source.require(byteCount) // Throws EOFException if insufficient bytes are available.
    return hex(source, byteCount.toInt())
  }

I'll revise what I said above @anuraaga has in the bench setup is fairly realistic as the bytebuffer isn't a cover for a byte array.
However, like Rag said, google's library is peeking and using Unsafe under the covers. I'm not sure Okio does that (or assumes it could).
Specifically, google's returns CodedInputStream.UnsafeDirectNioDecoder
  static CodedInputStream newInstance(ByteBuffer buf, boolean bufferIsImmutable) {
    if (buf.hasArray()) {
      return newInstance(buf.array(), buf.arrayOffset() + buf.position(), buf.remaining(), bufferIsImmutable);
    } else if (buf.isDirect() && CodedInputStream.UnsafeDirectNioDecoder.isSupported()) {
      return new CodedInputStream.UnsafeDirectNioDecoder(buf, bufferIsImmutable);
    } else {

Yup using Unsafe is probably the biggest difference and in the Armeria case it would end up in the Unsafe path if we so choose. From what I understand, wire / okio is mostly for Android, so it wouldn't make sense for them to implement support for Unsafe which doesn't work there. Protobuf had to support calling Java from C++... but interestingly the optimizations that went into doing that do apply to Netty-based servers too like here since they use direct memory.
I'm personally leaning towards not changing anything at this point though since for single digit gains, having two parser implementations seems like it's not worth it. What do you think?

yep.. if you revise from 10k spans per message down to 1k spans (which is a fairly high number anyway considering this is per host/second), there is not much difference between google and wire anyway. However, it does help memory wise to have wire decode bytes into hex strings.
with wire making hex manually
ProtoCodecBenchmarks.bytebuffer_protobufDecoder:bytebuffer_protobufDecoder·p0.999   sample            2716.557                us/op
ProtoCodecBenchmarks.bytebuffer_protobufDecoder:·gc.alloc.rate.norm                 sample     15  2271332.980 ±     49.158    B/op
ProtoCodecBenchmarks.bytebuffer_wireDecoder:bytebuffer_wireDecoder·p0.999           sample            3082.207                us/op
ProtoCodecBenchmarks.bytebuffer_wireDecoder:·gc.alloc.rate.norm                     sample     15  2707555.118 ±     43.790    B/op

with wire native hex stringing
ProtoCodecBenchmarks.bytebuffer_protobufDecoder:bytebuffer_protobufDecoder·p0.999   sample            2808.340                us/op
ProtoCodecBenchmarks.bytebuffer_protobufDecoder:·gc.alloc.rate.norm                 sample     15  2271332.557 ±     37.638    B/op
ProtoCodecBenchmarks.bytebuffer_wireDecoder:bytebuffer_wireDecoder·p0.999           sample            3045.130                us/op
ProtoCodecBenchmarks.bytebuffer_wireDecoder:·gc.alloc.rate.norm                     sample     15  2579549.017 ±     59.592    B/op

/me is done playing for now

@anuraaga leaning towards not changing anything I guess you mean changing the parsing approach? If so, I would say even if we wanted to, it can wait.
we are allocating more per request in parsing than necessary, but it is hard to tell the impact without looking at the total stuff allocated along the way. Regardless, I think we should merge your benchmarks so it can be picked up later.

PS @anuraaga I think the context you mention which is there are optimizations on the server that would be different than android-focused libraries is certainly spot-on. We shouldn't prevent ourselves from taking advantage of these optimizations considering this is server code.
I do remember some discussions last time about the guava sensitivity and heft of google libraries (ex. protobuf-java is 2MiB). I'm not super keen to re-enter that at the moment, but there could be some sense in using an unsafe mechanism such as they are.
At any rate, I learned thought this work that there's actually a measurable amount of gc time to recover. Eventhough I mentioned a little less priority, that's my personal 2p, if you are interested in taking the "attack the overhead" issue forward, please continue.

So in the interest of having these benchmarks and comparison points, made this PR non-draft, we may as well get it in and play with it some more.

Merging since this only touches anything in the not-so critical benchmarks module.


[FR] Add support for labels-only imports and merges

#1048 spurred some interesting discussion around the use case of importing labels.
To summarize, here are some proposed signatures for new methods under consideration:
Labels-only imports
Add new methods that enable creating datasets directly from labels files which implicitly include filepaths, without the need to provide a directory that contains both labels and actual media files:

Dataset.from_labels(labels_path, dataset_type, data_dir=None)
Dataset.add_labels(...)

An example use case here is creating a dataset, then exporting the labels for edit via an annotation provider, and then wanting to import the new labels back into FiftyOne.
Merge imports
Add the following methods:

Dataset.merge_dir(...)
Dataset.merge_importer(...)
Dataset.merge_labels(...)

which would basically be shorthand for import + merge operations of the following form:
tmp_dataset = fo.Dataset.from_dir(...)
dataset.merge_samples(tmp_dataset)

Supporting discussion
@ehofesmann
I'm trying to figure out the proper workflow for this entire process. When I was testing this, I took an image and uploaded it to CVAT and added new annotations. From there, I downloaded the new annotations file (and optionally the images). The annotation file has no reference to filepaths, only filenames. The issue comes in when trying to figure out the best way to import those new annotations. We don't have the option to import a dataset from_dir without an associated image directory. You could just move the downloaded annotations file to the same directory as your images, but then you would manually need to move the original labels.xml so you can add your new downloaded labels.xml to that dir. You could symlink/copy them to a new directory and easily import your dataset from_dir there, but then you need to use a key_fcn to match on fileNAME instead of path which sounds like it is inefficient.
In my mind, there are two ways to tackle this:

Allow importing datasets with only labels and then make it easier to match samples based on filename instead of filepath without requiring a key_fcn.
Update relevant importers to let you specify the data directory and labels files separately

I'm sure there are other approaches that may be better.
@brimoor
Thanks for sharing your workflow, it's definitely helpful to consider concrete use cases. I have a few thoughts here:

Regarding the desire to import labels without duplicating the media, I think https://github.com/voxel51/fiftyone/pull/1050 provides a helpful ingredient for that by supporting data.json files in place of actual media files, since you could now construct a data.json file for your to-be-imported dataset without interfering with the existing dataset's files that you're merging into. A next step could be to make the data.json file optional for dataset types where the labels file itself contains filenames or filepaths. The trouble here is that if the labels truly only contain filename with no directory, then there's not enough information there to create a new Sample, it would only be sufficient when merging using key_fcn = lambda sample: os.path.basename(sample.filepath). But, when combined with point 3 below, maybe that's okay.

Following on 1, perhaps we could add a Dataset.from_labels(labels_path, dataset_type) factory method, which would only support labeled dataset types where a single labels file contains all necessary information. Maybe it could have an optional data_dir argument which could specify a directory name in which all filenames in the labels files are presumed to reside.

We could add merge_dir() and merge_importer() functions to Dataset that would be similar to add_dir() and add_importer() but would add support for specifying a key_field or key_fcn that would be used to decide at-load-time whether to create a new Sample or merge the data into existing samples. This method could support arguments like insert_new that would provide the ability to specify that all data you're merging in should correspond to existing samples, and thus, for example, its not a problem if the labels file only contains filenames, because you would presumably have used a key_fcn that matches only based on filenames. In any case, the utility of merge_dir() would be to specifically serve use cases like importing edits without having to explicitly create a temporary dataset that is then thrown away. But, maybe it is more work than its worth, since addressing point 1 on its own would already support the workflow via a simple create + merge workflow in two steps.

@ehofesmann
I really like the idea of a Dataset.from_labels(labels_path, data_path, dataset_type), I do think that being able to specify a data_dir is the way to go (from_labels could also allow for a data.json I would think)
I agree that merge_dir() and merge_importer() are unnecessary at the moment.

As of fiftyone==0.12.0, iumporting and exporting labels-only variations of any dataset format for which that makes sense is now supported.
For example:
import os

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset(
    "open-images-v6",
    split="validation",
    label_types=["detections"],
    max_samples=10,
)

# Labels-only export
dataset.export(
    labels_path="/tmp/coco.json",
    dataset_type=fo.types.COCODetectionDataset,
    iscrowd="IsGroupOf",
    extra_attrs=False,
)

# The folder where all images live
data_path = os.path.dirname(dataset.first().filepath)

# Labels-only import
dataset2 = fo.Dataset.from_dir(
    data_path=data_path,
    labels_path="/tmp/coco.json",
    dataset_type=fo.types.COCODetectionDataset,
)

Also, a merge_dir() was added which combines from_dir() and merge_samples() allowing you to merge labels-only datasets into existing datasets with the same labels_path and data_path idea.


fix CI job name

Codecov Report
Base: 81.36% // Head: 9.65% // Decreases project coverage by -71.70% :warning:

Coverage data is based on head (8e40468) compared to base (e7d95d5).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #46       +/-   ##
==========================================
- Coverage   81.36%   9.65%   -71.71%
==========================================
  Files          12      12
  Lines        1191    1191
==========================================
- Hits          969     115      -854
- Misses        155    1072      +917
+ Partials       67       4       -63

Flag
Coverage Δ

go_tests
9.65% <ø> (ø)

mysql_tests
?

pg_tests
?

singlestore_tests
?

Flags with carried forward coverage won't be shown. Click here to find out more.

 Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


the facenet model

Would it be possible to get access to facenet_weights.h5 ? I am unable to find it anywhere, which make some parts of your code less useful.
I think you shared your gdrive link (https://drive.google.com/file/d/1971Xk5RwedbudGgTIrGAL4F7Aifu7id1/view?usp=sharing) - which does not seem work anymore.
Thanks

Google drive is out of the date source. Current one is here: https://github.com/serengil/deepface/blob/master/deepface/basemodels/Facenet.py#L1625

Hi Serengil,
Thanks for your quick help. I was able to download facenet_weights.h5. How Can I can specify this file along with this path to Facenet.loadModel() ?


Application release

Do I have to do something before publishing my application to the stores? if there are any restrictions to api that need to be removed?

Check that your app conforms to https://core.telegram.org/api/terms and that's all.


Can't run next code cell

Issue Type: Bug
Every now and then my notebook gets stuck executing a particular cell whenever I press shift+enter. It doesn't happen often but I think the steps are:

Run a cell (with shift+enter)
Add a next cell (after the first cell I ran)
Try to run the next cell (with shift+enter)

But the result of step 3 is that the first cell runs again.
My current workaround is to click into some other cell then back into the one I want to run. I'm not sure how often this works though, or if it's definitely what fixes it.
VS Code version: Code 1.60.1 (83bd43bc519d15e50c4272c6cf5c1479df196a4d, 2021-09-10T17:06:57.931Z)
OS version: Linux x64 5.11.0-34-generic snap
Restricted Mode: No

System Info

Item
Value

CPUs
AMD Ryzen 7 5800X 8-Core Processor (16 x 2200)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: disabled_offopengl: enabled_onrasterization: disabled_softwareskia_renderer: enabled_onvideo_decode: disabled_softwarevulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
2, 1, 1

Memory (System)
62.79GB (42.44GB free)

Process Argv
--no-sandbox --force-user-env --unity-launch --crash-reporter-id f337786d-50d8-4d4d-9348-e46ccd4b0f0d

Screen Reader
no

VM
0%

DESKTOP_SESSION
ubuntu

XDG_CURRENT_DESKTOP
Unity

XDG_SESSION_DESKTOP
ubuntu

XDG_SESSION_TYPE
x11

Extensions (18)

Extension
Author (truncated)
Version

gitlens
eam
11.6.0

gc-excelviewer
Gra
3.0.44

python
ms-
2021.9.1246542782

vscode-pylance
ms-
2021.9.2

jupyter
ms-
2021.8.2041215044

jupyter-keymap
ms-
1.0.0

remote-ssh
ms-
0.65.8

remote-ssh-edit
ms-
0.65.8

cpptools
ms-
1.6.0

vscode-paste-image
mus
1.0.4

java
red
0.82.0

vscodeintellicode
Vis
1.2.14

vscode-java-debug
vsc
0.35.0

vscode-java-dependency
vsc
0.18.7

vscode-java-pack
vsc
0.18.4

vscode-java-test
vsc
0.31.3

vscode-maven
vsc
0.33.0

markdown-all-in-one
yzh
3.4.0

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
pythonvspyt602:30300191
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt639:30300192
pythontb:30283811
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vstes263cf:30335440
pythondataviewer:30285071
pythonvsuse255:30340121
vscod805:30301674
pythonvspyt200:30340761
vscextlang:30333561
binariesv615:30325510
vsccppwtct:30364498
pythonvssor306:30344512
bridge0708:30335490
pygetstartedc2:30360494
bridge0723:30353136
pythonrunftest32:30365366
pythonf5test824:30361777
javagetstartedt:30364666
pythonvspyt187:30365361
pydsgst2:30361792
vssid140cf:30363604
vssur157:30367808

Probably https://github.com/microsoft/vscode/issues/133038 but I haven't been able to repro, would love reliable steps or a gif


(rest): fix listResource permission denied errorr.

Codecov Report

Merging #528 (17898e2) into release-1.2.0 (6927fbb) will decrease coverage by 0.00%.
The diff coverage is 0.00%.

:exclamation: Current head 17898e2 differs from pull request most recent head c8908eb. Consider uploading reports for the commit c8908eb to get more accurate results

@@                 Coverage Diff                 @@
##             release-1.2.0     #528      +/-   ##
===================================================
- Coverage            15.57%   15.56%   -0.01%
  Complexity             440      440
===================================================
  Files                  185      185
  Lines                 8675     8680       +5
  Branches               746      746
===================================================
  Hits                  1351     1351
- Misses                7206     7211       +5
  Partials               118      118

Impacted Files
Coverage Δ

...s/network/rpc/handler/ListResourcesURIHandler.java
9.09% <0.00%> (-0.59%)
:arrow_down:

...nk/wecross/restserver/fetcher/ResourceFetcher.java
6.55% <ø> (ø)

...main/java/com/webank/wecross/zone/ZoneManager.java
66.11% <0.00%> (-0.28%)
:arrow_down:


Add a quick test for decrypting openssl files

A quick example how to decrypt openssl files (and how to encrypt them if you follow bash script)
Adding this as a test so that we don't break anything going forward.

What is this testing that we weren't already testing before? We already test pbkdf2 in tests/pbkdf.rs and various forms of CBC in tests/save_restore.rs.

maybe it's useful as example then?, how to encrypt/decrypt to/from openssl's utility.
I agree that the code individually was tested properly.
Otherwise I'll drop the review.

Close this because long time no updates


New logo

@h2non I made this for the organization of jsdocs, I'm unable to access to the settings so could you upload it for me? Thanks

Will do tonight!

Thanks! I was afraid that you didn't like it because I'm conscious of my unusual type of design, but I'm glad you will do it, thanks.

Done! Due to Github CDN cache, it may take a couple of minutes to update the image.

Awesome, thanks!


[4.1][Feature][Ready] Allow for Custom Options, Such as minDate

A new inspection was created.

The inspection completed: No new issues

Wow - I think that's a very good way to pass configuration options to filters (through the options parameter) Great idea @zachweix ! I also agree to make it easy for developers to customize their date_range pickers.
We won't merge this as-is, though, because I think the syntax to do this is a bit off - and it's important to nail the syntax from day 1. Introducing breaking changes can only be done once every 6 months.
Here's what I mean by the that. Right now devs are supposed to do:
// daterange filter
$this->crud->addFilter([
  'type'  => 'date_range',
  'name'  => 'from_to',
  'label' => 'Date range'
],
  [
    'custom' => [
           'key' => 'value',
    ]
  ],
  function ($value) { // if the filter is active, apply these constraints
      // $dates = json_decode($value);
      // $this->crud->addClause('where', 'date', '>=', $dates->from);
      // $this->crud->addClause('where', 'date', '<=', $dates->to . ' 23:59:59');
  });

I think:

[ ] it would be better to use the entire options parameter for this; no custom; I don't see anything else we'd need to pass as options to this filter in the future, so let's use the entire parameter; that way, the syntax would be:

// daterange filter
$this->crud->addFilter([
  'type'  => 'date_range',
  'name'  => 'from_to',
  'label' => 'Date range'
],
  [ 'key' => 'value' ],
  function ($value) { // if the filter is active, apply these constraints
      // $dates = json_decode($value);
      // $this->crud->addClause('where', 'date', '>=', $dates->from);
      // $this->crud->addClause('where', 'date', '<=', $dates->to . ' 23:59:59');
  });

[ ] plus - possible problem: in the current implementation I don't think you can push boolean values, because they're echoed within quotes; instead of echoing them using {{ $key }}: "{{ $option }}", I would probably do something like this at the top of the filter:

  @php
     // make sure the jQuery plugin has the necessary configs to work
     // in case the developer hasn't specified any, use the defaults
     $filter->options = array_merge([
         'timePicker' => false,
	 'alwaysShowCalendars' => true,
	 'autoUpdateInput' => true,
     ], $filter->options);
  @endphp

Then inside the JS:
		jQuery(document).ready(function($) {
-			var dateRangeInput = $('#daterangepicker-{{ str_slug($filter->name) }}').daterangepicker({
+			var config = {
-				timePicker: false,
		        ranges: {
		            'Today': [moment().startOf('day'), moment().endOf('day')],
		            'Yesterday': [moment().subtract(1, 'days'), moment().subtract(1, 'days')],
		            'Last 7 Days': [moment().subtract(6, 'days'), moment()],
		            'Last 30 Days': [moment().subtract(29, 'days'), moment()],
		            'This Month': [moment().startOf('month'), moment().endOf('month')],
		            'Last Month': [moment().subtract(1, 'month').startOf('month'), moment().subtract(1, 'month').endOf('month')]
		        },
				@if ($filter->currentValue)
		        startDate: moment("{{ $start_date }}"),
		        endDate: moment("{{ $end_date }}"),
				@endif
-				alwaysShowCalendars: true,
-				autoUpdateInput: true
-			});
+			}.concat(@json($field->options));
+			var dateRangeInput = $('#daterangepicker-{{ str_slug($filter->name) }}').daterangepicker(config);

Notice that if we do it like this, we can also make a few more default options like timePicker, alwaysShowCalendars and autoUpdateInput configurable, with a default. Unfortunately, I don't think we can do this for ranges, startDate and endDate, since those depend on momentJS... So those will probably remain hardcoded and difficult to overwrite...
Let me know if you'd like to try the changes above yourself @zachweix , and modify your PR, or you'd rather wait for us to do them. It's good for us either way 😄

Ok so right after I posted the above I noticed something else 😄

[ ] if we pass the JS plugin options through the options parameter, we'd be introducing an inconsistency between how the date_range Filter and date_range Field work; so rather than passing the options through options, I think it would be better to follow the syntax of the date_range Field, and do:

// daterange filter
$this->crud->addFilter([
  'type'  => 'date_range',
  'name'  => 'from_to',
  'label' => 'Date range',
+ 'date_range_options' => [ 'key' => 'value' ],
],
  false,
  function ($value) { // if the filter is active, apply these constraints
      // $dates = json_decode($value);
      // $this->crud->addClause('where', 'date', '>=', $dates->from);
      // $this->crud->addClause('where', 'date', '<=', $dates->to . ' 23:59:59');
  });

It's a worse syntax. Definitely. But it would be consistent between Fields and Filters... which I think would be a good thing...
Eager to see what others think about this though...

just for reference: #2253

@zachweix thanks again for the PR. I've continued the work with my notes above, and it's now on a new branch here. Check out https://github.com/Laravel-Backpack/CRUD/pull/2500 please.


feat: EditorConfig icon

This adds an icon for .editorconfig.

Thanks! Do you mind updating light colour to one generated by script (https://github.com/nvim-tree/nvim-web-devicons/actions/runs/5880384728/job/15957851623?pr=294#step:10:27)?

Thanks! Do you mind updating light colour to one generated by script (https://github.com/nvim-tree/nvim-web-devicons/actions/runs/5880384728/job/15957851623?pr=294#step:10:27)?

done, I was unsure which color to pick.

Thanks!


Macros don't destructure terms

Passing the "same" input through two consecutive transforms can lead to surprising behaviour:
syntax m = ctx => {
  const term = ctx.expand('expr').value;
  return #`n ${term}`;
};

syntax n = ctx => {
  const [a,b,c] = ctx;
  return #`${a} ${b} ${c}`;
};

m 1 + 2
// expected: 1 + 2
// actual: Error: replacement values for syntax template must not be null or undefined

Now I know why this happens. The term is already built and Enforester::advance scoops it up and spits it out. But without knowing the implementation details this is unexpected.
Of course this works as expected:
syntax m = ctx => {
  const term = ctx.expand('expr').value;
  return #`n ${term}`;
};

syntax n = ctx => {
  const [term] = ctx;
  return #`${term}`;
};

m 1 + 2

I don't think this is a bug, but it could cause some confusion. I'm wondering what the impact would be on declarative macros.

I think the current behavior is "correct" as long as we're thinking of
these APIs as being low level power tools. When you work at this level you
must be aware that a context might return you a single "token" or a
partially enforested Term. Once the API stabilizes we need some good docs
to cover this point; you and I might be the only people who know about this
subtlety at the moment 😄
Once we start working on the high-level API we definitely need a way of
smoothing over this issue. The hazy idea I've had for a while is that
something like the match form #516 would be able to look into Terms and
match on flat tokens.
On Thu, Jan 26, 2017 at 8:11 PM Gabe Johnson<EMAIL_ADDRESS>wrote:

Passing the "same" input through two consecutive transforms can lead to
surprising behaviour:
syntax m = ctx => {
const term = ctx.expand('expr').value;
return #n ${term};
};
syntax n = ctx => {
const [a,b,c] = ctx;
return #${a} ${b} ${c};
};
m 1 + 2// expected: 1 + 2// actual: Error: replacement values for syntax template must not be null or undefined
Now I know why this happens. The term is already built and
Enforester::advance scoops it up and spits it out. But without knowing
the implementation details this is unexpected.
Of course this works as expected:
syntax m = ctx => {
const term = ctx.expand('expr').value;
return #n ${term};
};
syntax n = ctx => {
const [term] = ctx;
return #${term};
};
m 1 + 2
I don't think this is a bug, but it could cause some confusion. I don't
know what the impact would be on declarative macros.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/sweet-js/sweet.js/issues/636, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAJOhGSWp7GjUDZxV8uxtfkyNlY8abXks5rWW5ZgaJpZM4Lva0E
.

I can live w/ that. Maybe implement a Symbol method or two on Term. match/destructure.
👍  on the docs.


Short solution needed: "How to add volume to docker image" (docker)

Please help us write most modern and shortest code solution for this issue:
How to add volume to docker image (technology: docker)
Fast way
Just write the code solution in the comments.
Prefered way

Create pull request with a new code file inside inbox folder.
Don't forget to use comments to make solution explained.
Link to this issue in comments of pull request.

How to add volume to docker container


update 'plist'

Fixes https://www.npmjs.com/advisories/1650

This is clearly not being maintained anymore, and I'm no longer working with it either, so I'll close this PR. The fork and branch remain for anyone who wants to make use of it.


Numba nopython mode does not support numpy.cov

It seems that numba's nopython mode does not support numpy.cov
Here is a minimal reproducer.
from numba import jit
import numpy as np

@jit(nopython=True)
def test (x):
    y=np.random.randn(x,100000)
    return np.cov(y)

if __name__ == '__main__':

    print(test(5))

Thanks for the request. This was added in https://github.com/numba/numba/pull/3345 and will make it into the next release. If you'd like to try it out before hand try a development package from the numba channel on https://anaconda.org (but please wait a few hours to try so we can promote some new builds with the feature in!):
$conda install -c numba numba

Closing.


Exclude /node_modules/ from Chart.yaml searches

This is a strange PR and I would have no qualms with it being rejected.
Unfortunately excluding **/node_modules/** from vscode.workspace.findFiles(...) calls is somewhat necessary to prevent excessive CPU usage when this extension is installed and users open a large frontend project in VS Code. findFiles searches the entire repo by default, even .gitignore files.
One comment from a VS Code member recommends extensions exclude node_modules, since a proper fix to the findFiles API would be a breaking change: https://github.com/microsoft/vscode/issues/75314#issuecomment-503195666
This seems to be somewhat standard, even in extensions that have nothing to do with Node.js. For example:

Searching workspace.findFiles in the microsoft/vscode repo shows almost every usage excluding **/node_modules/** explicitly. https://github.com/microsoft/vscode/search?q=workspace.findFiles
This is the case for even the stock Markdown extension: https://github.com/microsoft/vscode/blob/c62f59fcae4d9fd029b542639f3ef58bd698f982/extensions/markdown-language-features/src/features/workspaceSymbolProvider.ts#L40

Would this be reasonable in vscode-kubernetes-tools as well?

Thanks to both of you for maintaining this extension and taking a look!


feat: adds new chart options to management, signal and dashboard

This PR aims to add a few new options to configure netbird components like:

signal protocol option between grpc, http and https
an option to change the signal port
changes signal log output to stdout instead of a file
adds a workaround to disable ipv6 directive (listen) from nginx (dashboard) conf

That workaround isn't the best approach, definitely, but I did it just for the sake of simplicity and to make the chart run.
A better approach, for example, would be having a configmap with nginx's default.conf snippet templatized to handle that specific ipv6 param (and maybe other params) properly, through the chart itself.

Hi, @jnodorp-jaconi just bumping up in case you did not receive the PR opened alert.

@diegocn FYI, I get this error from netbird-management after applying your changes:
panic: unexpected config protocol type grpc

I will change the default back to https. Missed that during review.

hey @jnodorp-jaconi
thanks for the feedback!


SublimeText 3, TabNine using up to 1Gb of RAM??

please complete the following information:

OS version: elementary OS 5.1.7
Editor: SublimeText
Editor version: 3.2.2
Programming language: Python
TabNine extension version: 2.9.0

If the issue is related to TabNine failure, please provide log files:
Type TabNine::config in your editor, the logs are located at the bottom of the page.
Issue Details:
The TabNine plugin is using up to 1 Gb of RAM. Is this expected?

Yes, this is expected.
By default, Tabnine deep local model is enabled, which requires roughly this amount of memory.
You can choose to disable it in tabnine::config - you will get completions that are not based on our deep model, the tradeoff being sacrifice quality to preserve memory.
Another option is to enable TabNine cloud if you choose (a paid option).


fix(run-job): update to POST instead of GET

See: https://github.com/dhis2/scheduler-app/pull/324

Closed in favour of the approach in #324


Re-enable RuntimePerfMetrics category

Metrics like average painted area, expensive events, etc were all disabled when timeline events got picked up from tracing, as per this bug.
Renable RuntimeperfMetrics, since the data obtained is very similar.

RuntimePerfMetrics make the following available - Layers, PaintedArea_total | PaintedArea_avg, NodePerLayout_avg,  ExpensivePaints, GCInsideAnimation, ExpensiveEventHandlers.
Of these, ExpensivePaints, PaintArea and NodesPerLayout can easily be calculate as they only need one event.
However, metrics like GCInsideAnimation and ExpensiveEvent handlers look for nested events, which don't seem to occur. Need to investigate how to calculate them.


2.0.2 data viz paso robles

Resolves #866 and Coda 178

@chachasikes Is this the expected display?


Support HCL colors

Is there any plan to support HCL color spaces? They are perceptually uniform and can be used to create color maps like these:

(from https://hclwizard.org/)
Ideally, a color map (such as Viridis) would first be initialized and then used it in a table to color the background of the cells according to their values.

for PDF the colors must at the end be converted into rgb or cmyk.  So if you want that, please provide a suitable converter function (with a reliable source as a reference).  See https://github.com/latex3/xcolor/issues/25 for an example.

According to https://www.easyrgb.com/en/math.php, it seems that a CIE-L*uv color can be converted to RGB via CIE XYZ as follows:
CIE-L*uv → XYZ
//Reference-X, Y and Z refer to specific illuminants and observers.
//Common reference values are available below in this same page.

var_Y = ( CIE-L* + 16 ) /116
if ( var_Y^3  > 0.008856 ) var_Y = var_Y^3
else                       var_Y = ( var_Y - 16 / 116 ) / 7.787

ref_U = ( 4 * Reference-X ) / ( Reference-X + ( 15 * Reference-Y ) + ( 3 * Reference-Z ) )
ref_V = ( 9 * Reference-Y ) / ( Reference-X + ( 15 * Reference-Y ) + ( 3 * Reference-Z ) )

var_U = CIE-u* / ( 13 * CIE-L* ) + ref_U
var_V = CIE-v* / ( 13 * CIE-L* ) + ref_V

Y = var_Y * 100
X =  - ( 9 * Y * var_U ) / ( ( var_U - 4 ) * var_V - var_U * var_V )
Z = ( 9 * Y - ( 15 * var_V * Y ) - ( var_V * X ) ) / ( 3 * var_V )

XYZ → Standard-RGB
//X, Y and Z input refer to a D65/2° standard illuminant.
//sR, sG and sB (standard RGB) output range = 0 ÷ 255

var_X = X / 100
var_Y = Y / 100
var_Z = Z / 100

var_R = var_X *  3.2406 + var_Y * -1.5372 + var_Z * -0.4986
var_G = var_X * -0.9689 + var_Y *  1.8758 + var_Z *  0.0415
var_B = var_X *  0.0557 + var_Y * -0.2040 + var_Z *  1.0570

if ( var_R > 0.0031308 ) var_R = 1.055 * ( var_R ^ ( 1 / 2.4 ) ) - 0.055
else                     var_R = 12.92 * var_R
if ( var_G > 0.0031308 ) var_G = 1.055 * ( var_G ^ ( 1 / 2.4 ) ) - 0.055
else                     var_G = 12.92 * var_G
if ( var_B > 0.0031308 ) var_B = 1.055 * ( var_B ^ ( 1 / 2.4 ) ) - 0.055
else                     var_B = 12.92 * var_B

sR = var_R * 255
sG = var_G * 255
sB = var_B * 255

XYZ (Tristimulus) Reference values of a perfect reflecting diffuser

Illuminant [Observer]
X2 [2° (CIE 1931)]
Y2 [2° (CIE 1931)]
Z2 [2° (CIE 1931)]
X10 [10° (CIE 1964)]
Y10 [10° (CIE 1964)]
Z10 [10° (CIE 1964)]
Note

A
109.850
100.000
35.585
111.144
100.000
35.200
Incandescent/tungsten

B
99.0927
100.000
85.313
99.178;
100.000
84.3493
Old direct sunlight at noon

C
98.074
100.000
118.232
97.285
100.000
116.145
Old daylight

D50
96.422
100.000
82.521
96.720
100.000
81.427
ICC profile PCS

D55
95.682
100.000
92.149
95.799
100.000
90.926
Mid-morning daylight

D65
95.047
100.000
108.883
94.811
100.000
107.304
Daylight, sRGB, Adobe-RGB

D75
94.972
100.000
122.638
94.416
100.000
120.641
North sky daylight

E
100.000
100.000
100.000
100.000
100.000
100.000
Equal energy

F1
92.834
100.000
103.665
94.791
100.000
103.191
Daylight Fluorescent

F2
99.187
100.000
67.395
103.280
100.000
69.026
Cool fluorescent

F3
103.754
100.000
49.861
108.968
100.000
51.965
White Fluorescent

F4
109.147
100.000
38.813
114.961
100.000
40.963
Warm White Fluorescent

F5
90.872
100.000
98.723
93.369
100.000
98.636
Daylight Fluorescent

F6
97.309
100.000
60.191
102.148
100.000
62.074
Lite White Fluorescent

F7
95.044
100.000
108.755
95.792
100.000
107.687
Daylight fluorescent, D65 simulator

F8
96.413
100.000
82.333
97.115
100.000
81.135
Sylvania F40, D50 simulator

F9
100.365
100.000
67.868
102.116
100.000
67.826
Cool White Fluorescent

F10
96.174
100.000
81.712
99.001
100.000
83.134
Ultralume 50, Philips TL85

F11
100.966
100.000
64.370
103.866
100.000
65.627
Ultralume 40, Philips TL84

F12
108.046
100.000
39.228
111.428
100.000
40.353
Ultralume 30, Philips TL83

Sorry but I don't have the time to write a converter function.  If you want it, please implement it and make a suitable pull request or at least a small exampl.e

Then I will close this issue for now. It can be reopened if someone offers some code.

@ttusar If you are using LuaTeX then you can use https://github.com/zauguin/luaicc for such maps.

I will have a look, thanks @zauguin for the pointer

To anyone else stumbling over this issue, there is now an example on Overleaf that builds on the code provided by @zauguin. Hopefully it will be of help to someone.


Issue #121 - Update call to timeout to account for breaking change

Fix for issue #121.

@mkaminski1988 can you take a look why the tests failed?
I wonder if we should run those tests on the image that is being used to run the plugin in (cloud-sdk), since the tool executes shell.

@msuterski I did fix the tests, which pass locally. I'm not sure why the tests break in drone, haven't dug into it yet.

@msuterski @mkaminski1988 I tried pushing these changes to a branch of non-fork (thought perhaps the fact that the source branch here was from a fork may have caused the failures) and now the tests are passing in CI (push | pull), but there is a new failure occurring in the slack step..


kernel won't respond to 'attach' debug message

Environment data
Version: 1.60.0-insider (user setup)
Commit: 8dbad73de575e47e127c8b1a4324f2b61257fbdc
Date: 2021-08-27T21:25:29.781Z
Electron: 13.1.8
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Windows_NT x64 10.0.19043
jupyter: v2021.8.2031173646
python: v2021.8.1159798656
Expected behaviour
Run by line starts
Actual behaviour
nothing happens, cell doesn't even run but the context key thinks we are in run by line
Steps to reproduce:

Unfortunately, there are no steps. Just play with run by line, until a cell doesn't run when clicking the RBL button.

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

logs.txt

Dupe of #7586
That one has a consistent repro


BOSH team admin cannot get task state after deployment deletion

The issue occurs when a user is a team admin (not bosh admin or bosh director admin), they make a deployment and subsequently attempt to delete that deployment. The user cannot then successfully get task state relating to the deletion of their deployment.
We are using BOSH release 255.4.
To reproduce:

Create team admin user in UAA
uaac client add $CLIENT_ID \
    --secret $CLIENT_SECRET \
    --authorized_grant_types "refresh_token password client_credentials" \
    --authorities bosh.teams.my-team.admin \

Deploy with that user
BOSH_CLIENT=$CLIENT_ID BOSH_CLIENT_SECRET=$CLIENT_SECRET bosh deploy

Delete the deployment
BOSH_CLIENT=$CLIENT_ID BOSH_CLIENT_SECRET=$CLIENT_SECRET bosh delete deployment $DEPLOYMENT_NAME

The last step does delete the deployment, but reports failure:
Acting as client 'kafka-on-demand-broker-dev1' on deployment 'kafka-on-demand-broker-dev1-team' on 'enablement-bosh-v2'

You are going to delete deployment `kafka-on-demand-broker-dev1-team'.

Are you sure? (type 'yes' to continue): yes

Director task 12969
  Started deleting instances > broker/0 (9d07caf6-e180-42b4-a379-3eccbf994be4)Error 600000: Require one of the scopes: bosh.admin, bosh.76f2d2b1-fc3f-47d2-bbfe-4109dc799337.admin

In addition, if we call the tasks API endpoint directly, we observe a change in the response when the task completes. This request:
curl -v --insecure --header "Authorization: Bearer $OAUTH_TOKEN" https://<redacted>:25555/tasks/12969

While the task is running, results in this response (edited for brevity):
< HTTP/1.1 200 OK
{"id":12969,"state":"processing","description":"delete deployment kafka-on-demand-broker-dev1-team","timestamp":1461083208,"result":null,"user":"kafka-on-demand-broker-dev1","deployment":"kafka-on-demand-broker-dev1-team"}

The same request immediately after the task has completed results in this response:
< HTTP/1.1 401 Unauthorized
{"code":600000,"description":"Require one of the scopes: bosh.admin, bosh.76f2d2b1-fc3f-47d2-bbfe-4109dc799337.admin"}

The OAuth token was obtained just before the deletion, and is valid for six hours, so this is not a result of token expiry.
Thanks,
@mariantalla and @tomwhoiscontrary

yup we've run into this bug. fix is via this story: https://www.pivotaltracker.com/story/show/117557245
will be in bosh v256.

Fixed in director v256.


update dutch timedelta

Coverage remained the same at 91.902% when pulling 0a5564c625dd641315f8609c3b87138b72b85709 on fix_dutch_timedelta into 9b9580eb769c20bee4bbf806595379d0387e6d1c on master.


HDFS-15856: Make recover the pipeline in same packet exceed times for…

… stream closed configurable.
@jojochuang Could you help review this?
Thanks.

I think the test failed is not related to this.

@jojochuang
Could you help check the latest patch?
Thanks.

Committed to trunk. Thanks @qizhu-lucas for your contributions. Thanks @ayushtkn and @jojochuang for your reviews.


Examples

I can't decode from the tests how to use this library. Currently I am not even sure how to create a track object. So ya, docs would be nice...

You won't use this directly, but rather through Segment. Docs are available for respective platforms here - https://segment.com/docs/


AP series term/sum problem

Function to generate problem to solve either sum of n terms of an AP or to calculate the n'th term of the AP

Sure
Thank you for the feedback


[bug][GB28181] does not reinvite after the client unexpectedly stops streaming

Describe the bug
When the client stops streaming without sending a BYE message (for example, when the camera is directly rebooted by power cycling), SRS (Streaming Relay Server) retains the old media connection and does not reinvite.
Version
5.0
To Reproduce
Described above
Expected behavior
Described above

Dup to #3944


Need R50+ViT-B_16 rather than R50-ViT-B_16!

please use wget https://storage.googleapis.com/vit_models/imagenet21k/R50+ViT-B_16.npz to get the correct pretrained model.

thanks


chore(gha): fix labeler

Pull Request
Purpose

Approach

Open Questions and Pre-Merge TODOs

Requirements

[ ] These changes meet the standards for contributing.
[ ] I have read the code of conduct.


Update mancy to 2.1.0

Thank you for the contribution. It was merged directly as 2b8a4957dfe47805503af1bc8e90fd15daf26f2b to keep commit history cleaner. Your contribution is still credited to you.


Full TX history (including incoming TXs)

Added the ability to fetch the entire history via etherscan and merge it with the local transactions.
This one is based off #32  so we might want to merge that one first

Codecov Report

Merging #33 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master    #33   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files          18     18
  Lines         908    929   +21
  Branches      101    102    +1
=====================================
+ Hits          908    929   +21

Impacted Files
Coverage Δ

src/TransactionController.ts
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e32ef21...5e4de93. Read the comment docs.


tutorial blog fix: remove non functional code

removed non imported remix json function returning post slug when creating new post.

Hi and thank you for taking the time to propose this PR 🙏🏼
The call you removed is returning the whole new Post object, not its slug only.
It's a good practice to return the newly created entity from there, to be able to do things like optimistic UI for example.
We are in the process of retiring the blog tutorial to use remix stacks, so we will leave the code as it's for now. The new tutorial will be up soon and we will be more than glad to have your feedback then.


Simplify editable interface

There should be an easy way to access row items. This is handy for digging id etc.

I guess this is obsolete now that I redid the API. Far simpler now. Data can be added as needed.


⚠ GitLab Server has degraded performance

In a14ae25, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 11193 ms

Resolved: GitLab Server performance has improved in e9e860a.


XfreeRDP Segfault with multimedia redirection

Attempting to use multimedia redirection with a Windows 2008 R2 Session Host server, however whenever the video is about to play xfreerdp crashes with a segfault. The videos play fine on Windows 7 fat clients as well as newer Wyse terminals supporting redirection.
Are there certain libraries or packages that need to be installed for multimedia redirection to work propertly with xfreedp?
Using Ubuntu 12.04 and latest Xfreerdp from Ubuntu Precise PPA (1360+32) as of today.
My command line is as follows
xfreerdp /v:tsfarm.lan.local /bpp:16 /d:domain.com /f +compression -sec-nla -encryption -wallpaper -window-drag -menu-anims -themes -decorations -sec-tls /multimedia /gdi:hw /cert-ignore
The issue also persists if I only use:
xfreerdp /v:tsfarm.lan.local /multimedia

Closing due to being inactive for a long time and redirection having received many fixes in the meantime. Repoen if still relevant.


fix: linter for log message

Description
Add a check for log messages to prevent them ending with punctuation.
Refs: https://github.com/stackrox/stackrox/pull/8625
Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added
[ ] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.
Testing Performed
Here I tell how I validated my change
make roxvet

Reminder for reviewers
In addition to reviewing code here, reviewers must also review testing and request further testing in case the
performed one does not seem sufficient. As a reviewer, you must not approve the change until you understand the
performed testing and you are satisfied with it.

As discussed offline, I think the linter should ignore multiline log statements like
Why?


getThreadInfo() error when chat room isn't named

when I call getThreadInfo() in a room that isn't named, it throw this error:
TypeError: Cannot read property 'name' of undefined

#296 uhmm..

I probably need to push this fix to npm.  Thanks @yuyuvn

@bsansouci did you say you were having problems with getUserInfo? If so, that could be causing the problems here and in #296.
We probably want to write some failsafe code to still return a sane result even if getUserInfo fails.

It's a bit weird since it works if I get the ID from getThreadInfo and then put the id into getUserInfo. In this case, the threadID and the userID used are the same, as expected.

Can you verify that these problems still occur with v1.2.0?

I'll check again

it work


Convert intersections to place objects with nodes.

My ideas for updating the intersections was to implement a less dynamic but more pluggable system, never got around to it though.
Quite a bit of the code complexity in the RA code base revolves around the creation of dynamic intersections.
Possibly converting intersections to their own placeable objects with node "connection points" that the RA road would hook into and adjust its tangent appropriately. This would reduce code complexity significantly and allow users to plug in their own intersections, on top of whatever base models would be provided.
This system would also allow editor placement of exits / onramps / parking lots to existing road systems. Where the connection type would be 'pass-through' rather than 'intersection' in the code base.
I think a system like that would allow quite a few possibilities that the current system doesn't.

Have you started implemented that?


Support for SQLite

As handled here, SQLite has been added to sqlectron-core, we should be able to support this for sqlectron-term as well.

SQLite support added also to sqelectron-gui. When you add sqlite bd to sqelectron-gui it appeares also in sqelectron-term, but get exception accessing it.


merge latest elixir syntax file from atom

I was wanting to apply syntax highlighting to function parameters and single/double quoted atoms (such as "quoted-atom":) which are now supported by the latest Elixir syntax file in Atom.
In merging the Atom file, I retained the recent commit https://github.com/fr1zle/vscode-elixir/commit/d905450864490f4fd5eb16932557b58b77199bc4 by @fr1zle which is the only code in this file that doesn't exactly track the Atom version.
Please let me know what you think and thanks for your work on this great extension!

LGTM, Thanks for contributing!

Thanks @fr1zle for the very fast review and merge, really appreciate it!  Hope you have a great weekend.


Add UI to inject sources in CSP

With a watcher to list subresources (potentially) used by a bookmarket

Shouldn't be required for Firefox 58+, where CSP is no longer applied to content scripts' subresources


Update get_kedro_project_json_data import path

Descriptions
Resolves https://github.com/kedro-org/vscode-kedro/issues/143
This pull request includes a change to the bundled/tool/lsp_server.py file to improve compatibility with different versions of kedro-viz. The most important change is the addition of a try-except block to handle imports for kedro-viz versions greater than and equal to 10.0.0.
Developer Note

bundled/tool/lsp_server.py: Added a try-except block to import get_kedro_project_json_data from the correct module based on the kedro-viz version.

@noklam

Is this still needed after #155?

Yes, as #155 updates the NPM and not pypi,  Also we need to update this from kedro_viz.api.rest.responses.pipelines import get_kedro_project_json_data as get_kedro_project_json_data moved to kedro_viz.api.rest.responses.pipelines

Would you bump the minimum version of kedro-viz to 10.1.0 after this?

I will update the minimum version of kedro-viz to 10.1.0 in this PR only

I suppose we will have to be careful abt making such changes on kedro-viz in the future as it is a breaking change on vscode.

@rashidakanchwala It is technically a breaking change since it's public API. It's ok to do the refactor but you may need to leave the import in the original place temporarily to keep it non-breaking until 11.0.0. i.e. keep this in kedro_viz.api.rest.responses.py
from .pipelines import get_kedro_project_json_data


⚠ Bofrost.lu has degraded performance

In e4dd840, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1286 ms

Resolved: Bofrost.lu performance has improved in 5a76699 after 7 minutes.


Added disabled and disabledValues prop/option

Added disabled prop that allows to enable/disable the picker.
Added disabledValues prop the hides the specified values.

this is a useful feature


EarlGrey is frequently crashing internally during edo_createChannelWithPort while very early in the first test

I believe this crash is caused from the recent update to EG, which creates the EDOHostService lazily.
Xcode reports: Thread 1: EXC_BREAKPOINT (code=1, subcode=0x1b8e18f14) on line 155 of EDOChannelPool.m

Did your app launch?

Using EG to interact with the device isn't supported if the app isn't launched? (ie. interact with system to modify OS settings before the app is launched, then launching app, then running tests)
I've run the test a few times since, but haven't seen this failure again. I think you're correct that I wasn't launching.  I'll reopen/ create a new ticket if I get more data that shows this doesn't appear to be the case. I'll close this ticket out as this doesn't appear to be a real issue.


Sorting session.

Actual Behaviour
The session are being displayed in an unsorted order.
Expected Behaviour
The session list  must be sorted on the basis of session date and time.
Screenshots of the issue

@the-dagger @mananwason @mariobehling
Should I send a PR?

@the-dagger we should also sort the tracks.
What do you think?


Migrate friction log template from google docs

Migrate friction log template from google docs
https://docs.google.com/document/d/1kcEv9c_diZbqdXk2VkVu6miHz62hUoxxzfIo5ILnESM/edit#

fyi @brainwane.  We've added this here, to be able to track these within the issue tracker :)  Maybe it's worth adding some documentation at the beginning to explain when/why they should be used?

Thanks! Indeed it might be -- I don't have time to put in the issue or add that documentation, though!


Pypi project Name is wrong

Hi,
First of all thanks, I going to look if I can help,
I don't know if it intended but the pypi name is :  elgeql-qb instead of edgeql-qb
A small fix to do !
thanks

Thank you for your interest in the project and for finding a typo in such a place.


Support configuration properties available as env vars in Docker

Is there any plan for flink docker image to support overrite configurations via env vars ?
I am using kubernetes to deploy a flink session cluster. When I want to change taskmanager.heap.size, I have to use Configmap to overrite flink-conf.yaml in container.
I would like to contribute the feature if the idea is approved.

This is something we've considered, but I would greatly prefer anything like this to be supported by Flink upstream first, instead of implementing it as an additional configuration method by the Docker images alone.
Flink already has some fragmentation with respect to how it's configured, and I'm trying hard not to have the Docker images contribute to it. You might consider searching for/opening a ticket on Apache Flink JIRA or starting a discussion on the user mailing list.

Thanks for your detailed answer and your advice.  :)


New Serializer API

Update for serde-rs/serde#437.

@erickt @oli-obk this is about 5-15% improvement in serialization speed.

Wow! I didn't expect it to be that significant. At this point might actually have the fastest json serializer. @dtolnay: how's the json benchmarks repo? Are you still working on it? If we can demonstratively show we are winning in most cases we should write this up and publicize it.

json-rust is faster on some inputs - see the latest numbers in this commit. And it is much faster on small inputs.

Awesome

Great work!