[dev] Integrate MobileSAM into the pipeline for lightweight and faster inference

Reference: https://github.com/ChaoningZhang/MobileSAM
Our project performs on par with the original SAM and keeps exactly the same pipeline as the original SAM except for a change on the image encode, therefore, it is easy to Integrate into any project.
MobileSAM is around 60 times smaller and around 50 times faster than original SAM, and it is around 7 times smaller and around 5 times faster than the concurrent FastSAM. The comparison of the whole pipeline is summarzed as follows:

Best Wishes,
Qiao

9e7533268c2990e9ec1cc7101643de8a6d5b8d84
Thanks for your good work. SAMM now supports mobile_sam


button that releases bubbles when clicked

closes #380

@hashfx
Looking forward to more contributions from you. 🧑💻

Thank you so much for your contribution.🌟
Make sure to ⭐ and follow me.

I would love to connect with you all ❤

https://www.linkedin.com/in/om-gawande/


Use Stream-Style Serialization in Container Registry

Description
Updates com.azure:azure-containers-containerregistry to use stream-style serialization instead of Jackson Databind.
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-containers-containerregistry

/azp run java - containerregistry - tests

/azp run java - containerregistry - tests

/azp run java - containerregistry - testsv

/azp run java - containerregistry - tests


Fix TypeScript not able to resolve types when set to Node16

Fix for this error when using TypeScript 4.7 set to Node16 module resolution.

Coverage remained the same at 97.903% when pulling 38a722f30298fda23435736a7aa5d5641fb10061 on Baune8D:fix-ts-ecma-resolving into eae1d8d8baed3e71dc2c1cdc81605b7d36e0a1cd on open-cli-tools:main.

Published as v7.2.2 ✅


Add NginxSidecarPatch template

There is a generic Sidecar patch. It can be used for nginx or other purposes


File upload example.

Please a file upload example.

I solved it with this library "apollo-upload-angular-link-http"


Configuration to let malware retrieve a HTTP response

Is there any configuration in Fakenet-NG that is similar to INetSim 's Fake Mode?
Description:
Delivers configured fake files based on the file extension in the HTTP request (e.g. .html or .exe) or static paths

Fakenet-NG already supports this for the following file types based on mime type: text/html, image/png, image/ico, image/jpeg, application/octet-stream, application/x-msdownload, application/pdf, application/xml (
see https://github.com/fireeye/flare-fakenet-ng/blob/master/fakenet/listeners/HTTPListener.py#L18). Does this feature not work for you?


PDFs with Active Hyperlinked Images - Error/Issues

Describe the bug in current situation
When PDF files with hyperlinked images are uploaded to the Records tab, and subsequently available in the Redaction App, several issues are occurring.
See MAG-2023-32691 in PROD as example.
For example, these two small images are hyperlinked to take them to their respective pages.
Private Zenhub Image
The first issue is the hyperlinked image is creating an annotation in the Comments tab (see image below, circled in yellow). In addition, a yellow paperclip is displayed on the page. The cannot be deleted or removed.
Second, when trying to produce the redline, the creation is stuck in progress, and likely stalled due to an error (presumably) related to this annotation.
Private Zenhub Image

Note that files that are converted to PDF through our conversion engine are flattened, and this issue won't occur on those files. This issue is primarily where a PDF is uploaded originally, with hyperlinked images enabled in the PDF.
Link bug to the User Story
Impact of this bug
Describe the impact, i.e. what the impact is, and number of users impacted.
Chance of Occurring (high/medium/low/very low)
Pre Conditions: which Env, any pre-requesites or assumptions to execute steps?
PROD and TEST
Steps to Reproduce
Steps to reproduce the behavior:

Upload a PDF file with hyperlinked images to the Records tab
Enter the Redaction App
See unknown annotation in Comments pane (yellow paperclip)
Next, flag all pages with an acceptable page flag, and attempt to produce the redline
See error thrown in Console, and creation is stuck in progress

Actual/ observed behaviour/ results
Unable to produce redline or delete annotation
Expected behaviour
A clear and concise description of what you expected to happen. Use the gherking language.
Screenshots/ Visual Reference/ Source
If applicable, add screenshots to help explain your problem. You an use screengrab.

The first issue is already captured as know issue and created a ticket (#4708) for the same. We can take a look at the second issue based on priority.

@KyEggleston - Could you please share a sample pdf with above issues? Is there any request to refer in test?

Still trying to get that sample file, as it's no longer on the request.

@divyav-aot  - Disregard the message above. The file in question is now on MAG-2023-07867 in TEST. When done investigating, if you can wipe this file, and any associated harms, redline or response package, that would be fantastic!

@divyav-aot - can you provide an estimate on this when you have a chance? Thanks!

@m-prodan - this is handling as part of #4708. So there is no separate effort for this ticket.

This cannot be tested with normal pdfs. The issue was specific to the prod file with paper clip in it. You can test pdf files with paper clip in it and see if you are getting the error in doc reviewer.

Annotation comment on the main pane shows Legend labeled comment instead of the annotation in the original document

original doc is

@divyav-aot could you please check?


Add missing stubs for brew snippets and master pages (Master Pages Pt 1)

This adds some missing bits for snippets stubs and adds the same for master pages.
I chose the name the field templates as it may be a better term for the user base than master pages when it comes to intent of the feature.
Updates tests to account for the helper function returning the template stub.
In addition to providing stubs, this provides direct support for #2804 and #3022.

Just FYI on this, I am going to put this on lower priority until we decide if Master pages are a feature we actually want to pursue yet. Just given the size of this feature, I don't want to get myself too involved in reviews until the discussion in #3783 comes to a conclusion on both need and approach.


Finding a smarter way to choose dates for events

Because November 31 doesn't exist LOL

Try JQuery datepicker? This might actually be my job...


Multi-entry setup

Hello,
Is there a support for multi-entry html?
Right now I have multiple HtmlWebpackPlugin instances for each html file:
    plugins: [
      new MiniCssExtractPlugin({
        filename: '[name].css',
        chunkFilename: "[id].css"
      }),
      ...inputFiles.map(path => new HtmlWebpackPlugin({
        inject: true,
        chunks: [path],
        filename: `${path}.html`,
        template: `src/${path}.html`,
      })),
      new HTMLInlineCSSWebpackPlugin({
        leaveCSSFile: false,
      }),

But the injected CSS is not right because all html files contains all css tyles.

Hi @icl7126 , can you provide a minimal demo that can produce this issue?

MultiEntryDemo.zip
Steps to reproduce:

extract zip
npm i
npm run w
check new folder target - it contains two folders with two html files - both files contains all css styles from both css files
Also the "" is still present before the actual inline style.

Hi @icl7126 , I fixed the issue at 1.7.0. Let me know if there is any problem

@Runjuu Thank you!!! It works perfectly now!
Also it's nice to see I'm not the only one programming on Sunday :)


Crust Grants Program

https://github.com/crustio/Crust-Grants-Program

Anyone is welcome to apply for a grant. Currently, Crust Grants program mainly targets projects of protocol layer, and applications that serve a wide range of users.
Generally, your project will have better chances to get accepted if:
It presents a well-researched or tested concept, for which ideally you are able to show some prior work.
You can demonstrate that the project will be maintained after completion of the grant, be it through an obvious commitment to the technology from your side, additional funding sources or an existing business model.
Your team has proven experience with the relevant languages and technologies and/or a strong technical background. We ask you to provide the GitHub profiles of all team members as part of your application and will examine these for past activity and code quality. Naturally, you can also link to projects on other platforms.
Your application is rich in technical details and well-defined.
You can clearly present how your project stands out among competitors or implements technology that doesn't exist in the ecosystem yet.
Additionally, it must fulfill the following requirements:
All code produced as part of a grant must be open-sourced, and it must also not rely on closed-source software for full functionality. We prefer Apache 2.0, but GPLv3 or Unlicense are also acceptable.
We do not award grants for projects that have been the object of a successful token sale.
Lastly, we do not fund projects that actively encourage gambling, illicit trade, money laundering or criminal activities in general.
In addition to the information provided on your application, note that your project will need to comply with our Guidelines for Milestone Deliverables. In particular, we require all projects to create documentation that explains how their project works. At a minimum, written documentation is required for funding. Tutorials or videos are also helpful for new users to understand how to use your product. If your team's background is purely technical, consider hiring a freelance or part-time technical writer as part of the grant.
Project Ideas
An overview of existing projects in the Crust Technology Stack along with broad project ideas we would potentially be interested in funding can be found here.
If you have a good concept of the technical challenges that your idea entails and would like feedback before applying, you can send us an<EMAIL_ADDRESS>and tell us about it.


Example with NEMO on multiple models and timeseries

Example with NEMO data using in composite models.
Statsmodels STLForecast implementation for ARIMA as a strategy.

Hello @ChrisLisbon! Thanks for opening this PR.


Suggestion: Transition to Next.js for Enhanced Performance and Firebase Integration

Hi Team,
I noticed the project is currently using separate frontend and backend folders, with React for the frontend. I’d like to suggest switching to Next.js, as it allows for building both the frontend and backend in a unified framework. Here's why Next.js could be a better fit:

Fullstack Capabilities: With Next.js, you can handle both frontend and backend logic, as it includes built-in API routes. This eliminates the need to maintain separate folders for the frontend and backend, streamlining development.

Server-Side Rendering (SSR) & Static Generation: Next.js supports SSR and static generation out of the box. This can drastically improve performance and SEO, especially beneficial for content-heavy or dynamic pages.

Automatic Code Splitting: Next.js automatically splits the code, making sure only necessary components are loaded initially, improving page load times.

Improved Developer Experience: The framework includes features like file-based routing and seamless TypeScript support, making development faster and more efficient.

I also noticed that the Firebase label is added but not yet used. Next.js integrates smoothly with Firebase for features like authentication, database, and hosting, making it easy to manage backend services within the same project.
Let me know your thoughts on this suggestion. I'd be happy to help with the migration if you decide to move forward with Next.js. Additionally, it would be great if you could add labels for Hacktoberfest and GSSoC Level 3 to this issue.

@VaibhavKatariya ,
Thank you for your suggestion! However, this project’s tech stack is based on React.js and is designed to be open for all contributors. Switching to Next.js at this stage could introduce challenges for contributors already familiar with React. also its against rule of hactoborfest and GSSoC
If you're interested in contributing with React, we’d be happy to welcome your involvement! Please feel free to reach out if you have any questions or if you'd like to work on React-based issues.

Hi @skmirajulislam ,
Thank you for the clarification! I understand the importance of keeping the project within the planned tech stack and the challenges a migration might introduce, especially considering Hacktoberfest and GSSoC rules.
I’d still love to contribute with React.js and will look for opportunities where I can assist. Please feel free to assign me any React-related tasks or issues that need attention!
Looking forward to keep contributing!
Best regards,
@VaibhavKatariya

@VaibhavKatariya
you can implement our backed, till now its empty for this ill give you label 3 under GSSOC & hactoberfest, if you able to implement it without any internal conflict, feel free to ask me!

@skmirajulislam,
I'm definitely interested in developing the backend, especially since I have hands-on experience with Firebase.
Could you provide a brief overview of what the backend should look like and what technologies we will be using? That would be highly appreciated!
Looking forward to your guidance!


Deprecated Exclusion Zones

Fixes #693

Thanks for the pull request @joseph-kaile!

:x: Missing CLA.

Please send in a Contributor License Agreement (CLA) and comment back here to let us know to check this!
Maintainers, here's how to check for a submitted CLA.

:grey_question: CHANGES.md was not updated.

If this change updates the public API in any way, please add a bullet point to CHANGES.md.

Reviewers, don't forget to make sure that:

[ ] Cesium for Unreal Samples works.

CHANGES.md, then I'll merge this one too.


enhancement request: Limit drafts more aggressively.

This is not an approved issue to work on yet, but if it resonates with you, discuss it on chat.
There's some strong sentiment among a couple users (@borisyankov and myself, for example) that we store way too many drafts, when you never really go back more than 3 or 4 drafts.
It would be nice to either just limit drafts more aggressively (and let power users like Tim deal with the rare cases that they lose their 6th to last draft or whatever) or make it a user-based setting.
Currently we limit drafts to the last 30 days.  It's possible that calendar time is inferior to just keeping a max count--you'll have to debate that on chat.

Thanks for the suggestion, @showell ! We don't have current plans to work on this.


Non-visible payment methods

There needs to be a section addressing what to do if I cannot see a payment method.
I see a notification indicating this, but I want to do something about it - I cannot delete it, nor change billing to use a newly added card.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 7d14e33d-0d3e-45f4-fefe-dec3ae7d351f
Version Independent ID: aeb53e2c-d35c-73fe-26f2-4f101bfeeea7
Content: Manage payment methods
Content Source: microsoft-365/commerce/billing-and-payments/manage-payment-methods.md
Service: microsoft-365-business
GitHub Login: @cmcatee-MSFT
Microsoft Alias: cmcatee

@redmeades-ud Thank you for your feedback. Ideally, if you don't see the payment method, we recommend contacting support.  We have added that information to the document via PR#https://github.com/MicrosoftDocs/microsoft-365-docs/pull/12431
Once the author approves, the changes will be live.
Thanks
Sri

@redmeades-ud Are you the admin who added the payment method you're trying to change/replace? What is the exact notification you see?

Closing since there's been no response from the customer on this.


Include auth_ext static assets in the package build

I verified that the assets are in the sdist.
Also add a test that fetches one of the assets. I'll kick off a
release dry-run to verify that it works.

Cool! Test makes sense to me, but does that really test that it is packaged?

Tests run on production packages, so yes.

Cool! Test makes sense to me, but does that really test that it is packaged?

Tests run on production packages, so yes.

To clarify, release/nightly/dry-run build tests run on production packages, and I kicked off a dry-run to test this, which looks good (all the failed tests are parser related and will be fixed by https://github.com/edgedb/edgedb/pull/6375): https://github.com/edgedb/edgedb/actions/runs/6672556465/job/18137460047
Running tests locally (and I think in CI) use the dev build.

Ahh, that was the piece I was missing to tie together the dry run and this test 🙌


jsfiddle examples are not opening with the provided link https://jsfiddle.net/t421d7cg/

Version

Vue version:

Description

Demo
Please use our JSFiddle template to reproduce the bug. Issues without working reproduction might be ignored.

The ones I've found are working.


Validate a document in React Native

Has anybody found a way of using the validation defined in AstroClass in the React Native?

Astronomy is tightly integrated with Meteor. So it might be quite difficult to use only validation. However, it's heavily modularized so in theory it's possible to strip it of the modules dependent on Meteor. In near future I will probably prepare npm Astronomy version to work with GraphQL. But it's a plan for now.

@lukejagodzinski How about the paln to partially move Astronomy to npm and leave only Meteor dependent parts in meteor package? I could/want help with this.

So far haven't moved with it far. I'm still trying to figure out what part
of astronomy I want to focus on when moving to npm. Schema validation or
database operations or both. In both cases I want it to work with GraphQL
but also other data layers. So I have to do more project in GQL to realize
what's the good thing. When I started working on Astronomy it was just
because I was solbing a problem with lack of the good model layer for
Meteor. So far, when I work with GQL there is a little need for tool like
Astronomy but myabe that's because all projects that I work on are not
super big
On Wed, Aug 15, 2018, 10:59 AM Kostya Kurilov<EMAIL_ADDRESS>wrote:

@lukejagodzinski https://github.com/lukejagodzinski How about the paln
to partially move Astronomy to npm and leave only Meteor dependent parts in
meteor package? I could/want help with this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jagi/meteor-astronomy/issues/607#issuecomment-413137138,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABAKXcfgMickP9kmDOyDtiuBGsjyloTeks5uQ-LugaJpZM4NL3UO
.


PF-2972: Sync client and service Spring Boot versions.

Jira: PF-2972 (continued)
What: A previous PR bumped Spring version forward to address a vulnerability for the service, but missed that the client version is also explicitly set.  This PR fixes the client too.
Why: Client's sole Spring Boot dependency (spring-boot-starter-json) does not have vulnerability at this version, but it is locking downstream consumers of the client to a Spring Boot version that causes them to pull vulnerable version of other Spring Boot libraries.  Also, having the client and server Spring Boot versions diverge seems bad.
How: Move the springBootVersion variable up to the top level project and consume it in both client and service sub-projects so that they stay in sync (and at the safe version).

@DataBiosphere/terra-ecm-code-owners : gentle bump on this, as it's blocking some upstream remediations, TIA


Update dpg version

https://github.com/Azure/azure-sdk-for-python/issues/24342

/check-enforcer override


[resotocore][feat] Allow metadata for kinds.

Description

Allow defining metadata as part of the resource class
Allow getting a flattened model
Add icon and group to all base resources

To-Dos

[x] Add test coverage for new or updated functionality
[x] Lint and test with tox

Code of Conduct
By submitting this pull request, I agree to follow the code of conduct.

Codecov Report
Patch coverage: 76.86% and project coverage change: -0.15 :warning:

Comparison is base (cfbe53f) 89.62% compared to head (28c2414) 89.48%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1505      +/-   ##
==========================================
- Coverage   89.62%   89.48%   -0.15%
==========================================
  Files          74       75       +1
  Lines       11827    11963     +136
==========================================
+ Hits        10600    10705     +105
- Misses       1227     1258      +31

Flag
Coverage Δ

resotocore
89.48% <76.86%> (-0.15%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

resotocore/resotocore/model/json_schema.py
70.65% <70.65%> (ø)

resotocore/resotocore/web/api.py
69.39% <75.00%> (-0.04%)
:arrow_down:

resotocore/resotocore/model/model.py
92.52% <96.66%> (+0.12%)
:arrow_up:

... and 2 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


User setting page responsive design

Update the user settings page to a responsive layout to solve the mobile display problem.

ref #363


Add platform information to User-Agent

Closes #361

(@pg9182 assured me he tested on client on Linux as well)


Heap profiler occasionally hangs for long periods of time on Node10 due to a GC bug in V8

On Node10 e2e tests, we've seen some failures which seem to be due to long pauses when collecting profiles in Node10.
This happens consistently. Below I show some of the pauses in logs seen on the continuous tests.
Today's 7am test (node10) had a ~7 minute gap between starting profile creating and getting created profile:
...
Tue May 15 14:08:45 UTC 2018: DEBUG:@google-cloud/profiler: Attempting to create profile.
Tue May 15 14:16:13 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.
...

Today's 5am  test (node10) had a ~7 minute gap between getting a profile after creation and collecting the profile, and another equally long gap between starting profile creation and getting the created profile:
...
Tue May 15 12:08:51 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.
Tue May 15 12:15:11 UTC 2018: DEBUG:@google-cloud/profiler: Successfully collected profile WALL.
...
Tue May 15 12:16:52 UTC 2018: DEBUG:@google-cloud/profiler: Attempting to create profile.
Tue May 15 12:22:45 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.
...

The 3am run had a ~11 minute gap between starting profile creation and getting created profile:
...
Tue May 15 10:08:46 UTC 2018: DEBUG:@google-cloud/profiler: Attempting to create profile.
Tue May 15 10:19:48 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.
...

The 1am run had a ~15 minute gap between starting profile creation and getting created profile:
...
Tue May 15 08:08:59 UTC 2018: DEBUG:@google-cloud/profiler: Attempting to create profile.
Tue May 15 08:23:18 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.
...

The 11am run had a ~10 minute gap between getting created profile and collecting the profile:
...
Tue May 15 06:08:45 UTC 2018: DEBUG:@google-cloud/profiler: Successfully created profile WALL.
Tue May 15 06:19:24 UTC 2018: DEBUG:@google-cloud/profiler: Successfully collected profile WALL.
...

I'm inclined to believe that the long pauses occur almost randomly with respect to what the profiler is doing (but show up most frequently between log statements that typically have more time in between them).
Will work on reproducing locally. If that doesn't work, then I'll try reproducing on a GCE instance with minimal resources. Then I want to see if this is more associated with the heap or time profiler.

Is this specific to Node 10?

Yes, this is specific to Node10.
No flakes like this have been seen in Node 6, 8, or 9, and looking there have not been any long periods between log statements (>1min) observed in Node 6, 8, or 9.

It would be good if you can add the following command line flags and
retrieve the node_trace.*.log files: --trace-events-enabled --trace-event-categories v8. It would be curious to see if anything
interesting shows up on the trace timeline.
Note that these flags would work on Node 8+ only.
On Tue, May 15, 2018 at 11:12 AM Margaret Nolan<EMAIL_ADDRESS>wrote:

Yes, this is specific to Node10.
No flakes like this have been seen in Node 6, 8, or 9, and looking there
have not been any long periods between log statements (>1min) observed in
Node 6, 8, or 9.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199#issuecomment-389262422,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAE0qaIgAjSQlBCkIWwQKCVgpF2Huxtbks5tyxqcgaJpZM4T_8bl
.

--
Ali

Can reproduce on a GCE instance.

Here's logging from a run. Trace is attached.
Warning: Trace event is an experimental feature and could change at any time.
[[21:32:15.330]] [LOG]   DEBUG:@google-cloud/profiler: Attempting to create profile.
[[21:33:18.912]] [LOG]   DEBUG:@google-cloud/profiler: Successfully created profile WALL.
[[21:33:29.552]] [LOG]   DEBUG:@google-cloud/profiler: Successfully collected profile WALL.
[[21:33:30.132]] [LOG]   DEBUG:@google-cloud/profiler: Successfully uploaded profile WALL.
[[21:33:30.133]] [LOG]   DEBUG:@google-cloud/profiler: Attempting to create profile.
[[21:33:38.622]] [LOG]   DEBUG:@google-cloud/profiler: Successfully created profile HEAP.
[[21:33:38.918]] [LOG]   DEBUG:@google-cloud/profiler: Successfully collected profile HEAP.
[[21:33:39.498]] [LOG]   DEBUG:@google-cloud/profiler: Successfully uploaded profile HEAP.
[[21:33:39.499]] [LOG]   DEBUG:@google-cloud/profiler: Attempting to create profile.
[[21:52:49.497]] [LOG]   DEBUG:@google-cloud/profiler: Successfully created profile HEAP.
[[21:52:49.502]] [LOG]   DEBUG:@google-cloud/profiler: Successfully collected profile HEAP.
[[21:52:49.798]] [LOG]   DEBUG:@google-cloud/profiler: Successfully uploaded profile HEAP.

node_trace.1.log
Log pause in output started at ~1.4min into the program, and ended at ~20.5 min in. So pause may correspond to V8.GCFinalizeMCReduceMemory

That shoudn't happen. My guess would be that one of the worker threads is descheduled and we are waiting for it or memory is swapped out.
Could you please re-run with --trace-event-categories v8,disabled-by-default-v8.gc and with --trace-gc --trace-gc-nvp flags?

@ulan I re-ran with both those flags.
I've attached the output and the trace from this run.
node_trace.2.log
out.txt

Thanks. The GC is stuck in V8.GC_HEAP_EXTERNAL_WEAK_GLOBAL_HANDLES.
This is the phase where the finalizers and phantom callbacks of weak persistent handles are called.
https://cs.chromium.org/chromium/src/v8/src/heap/heap.cc?rcl=1caa6178e4dfa30e075789831c0bacdc648844b7&l=1777
These callbacks are installed by NodeJS using the SetWeak API:
https://cs.chromium.org/chromium/src/v8/include/v8.h?rcl=1caa6178e4dfa30e075789831c0bacdc648844b7&l=628
The next step would be to instrument GlobalHandles::PostGarbageCollectionProcessing, GlobalHandles::PostMarkSweepProcessing, GlobalHandles::DispatchPendingPhantomCallbacks to find which callback is taking long time.

@nolanmar511 are you able to reproduce this locally at all?
Secondly, are you able to reproduce this with the Node.js master branch / canary build? If we are going to add more instrumentation, we will need to run with our own build.
@ulan: can we add permanent instrumentation (i.e. trace events - disabled by default) to the above locations. These sound like locations where we should have probes as they sound useful for debugging in the future as well. WDYT?

@ofrobots It was mentioned that it's reproducible on a GCE instance but not locally. @nolanmar511 It would be good to look at what the CPU usage looks like when this happens (is the node process idle or it's doing something CPU intensive). It would also be good to see if this reproduces with just CPU profiler on or just heap profiler on.

@ofrobots -- I'm currently working on a g1-small GCE instance.
I can reproduce with v11.0.0-v8-canary20180516056b78fafe.
Also, CPU use is high (greater than 90%) during the long pauses.

@ofrobots adding a trace-event per callback might be too expensive. We would need to measure this on benchmarks.
Adding high-level trace-event to GlobalHandles::* functions sounds good to me.

I should try setting a ulimit to see if I can reproduce this locally.

Also, I should run and collect profiles with gperftools.

I collected the attached profile using node --prof.
isolate-0x33671d0-v8.log
Not sure how to add more instrumentation (or what instrumentation should be added). Building on a GCE instance is slow, but doable; copying a built version of node to a gce instance could also work.

Still working on running with gperftools (using CPPFLAGS=-g nvm install -s 10.1.0 to install node 10 on the GCE instance is slow; it failed the first time).

@nolanmar511 But are you still able to reproduce the issue with that modified VM?

This is related to having the heap profiler enabled.
I ran with only the CPU profiler enabled, and had no problems. Running with the heap profiler only, I was able to reproduce this.

Was able to reproduce with a build of node cloned from https://github.com/nodejs/node.
Working on adding more tracing now.

I collected a trace with the version of node I'd built.... it's hsowing ~10 minutes spent in V8:DispatchPendingPhantomCalls.
node_trace.2.log

Since this happens in V8:DispatchPendingPhantomCalls and only with the heap profiler enabled, it could be the SamplingHeapProfiler::OnWeakCallback:
https://cs.chromium.org/chromium/src/v8/src/profiler/sampling-heap-profiler.cc?q=SamplingHeapProfiler::OnWeakCallback&sq=package:chromium&dr=CSs&l=104
Does it have O(n) complexity in the number of samples_? If so, then the total complexity of calling the phantom callbacks would be O(n^2), which would explain the long pause.

@ofrobots PTAL

@nolanmar511 can you provide the post-processed version of the --prof log you included in https://github.com/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199#issuecomment-391184462. To precise same version of node used to generate the log is needed to process the log (using --prof-process).
@ulan: Has anything changed in V8 in Node 10 that we would start seeing this pathology. We take a sample every 512KiB, so even if we had a 2GiB heap, we should have around 4096 samples on average. This would not explain the 19 minute pause.
Let's see what the --prof CPU profile shows. If something is looping for 19 minutes, it should show up in the profile. BTW @aalexand this is one of the reasons why having native symbols available in the CPU profile would be quite helpful.

@ofrobots -- I've been using node v10.1.0

The --prof profile doesn't seem very useful:
 [C++]:
   ticks  total  nonlib   name

 [Summary]:
   ticks  total  nonlib   name
  37501   27.4%   95.0%  JavaScript
      0    0.0%    0.0%  C++
  96580   70.5%  244.7%  GC
  97583   71.2%          Shared libraries
   1967    1.4%          Unaccounted

 [C++ entry points]:
   ticks    cpp   total   name

 [Bottom up (heavy) profile]:
  Note: percentage shows a share of a particular caller in the total
  amount of its parent calls.
  Callers occupying less than 1.0% are not shown.
   ticks parent  name
  85603   62.5%  /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22

  15741   11.5%  LazyCompile: *busyLoop /home/nolanmar/testing/busybench.js:26:18
  13363    9.8%  Builtin: KeyedLoadIC_Megamorphic
  13122   98.2%    LazyCompile: *busyLoop /home/nolanmar/testing/busybench.js:26:18

  11199    8.2%  /home/nolanmar/.nvm/versions/node/v10.1.0/bin/node
    677    6.0%    /home/nolanmar/.nvm/versions/node/v10.1.0/bin/node
    338   49.9%      LazyCompile: *busyLoop /home/nolanmar/testing/busybench.js:26:18
     92   13.6%      LazyCompile: ~Script vm.js:38:14
     92  100.0%        LazyCompile: ~createScript vm.js:245:22
     92  100.0%          LazyCompile: ~runInThisContext vm.js:294:26
     92  100.0%            LazyCompile: ~Module._compile internal/modules/cjs/loader.js:639:37

Most of the ticks are in libstdc++ but precise symbol is not available.

Can you give me instructions to reproduce this myself?

Here's a flamegraph: https://gist.github.com/ofrobots/dbc55eb6ab27ea06be247384d86af5c6 (download and open). We are indeed spending a lot of time in the STL tree implementation. I am investigating.

The set of samples indeed grows to a really large size (~200k) before the weak callback is called, and the weak callback is quadratic in complexity indeed. It seems that something has changed in GC that delays the weak-callbacks from being called incrementally resulting in this massive pause. All but 200 of the ~200k samples get their weak callback called in one shot.
I can reproduce this behavior on a workstation too.

@ofrobots and I tracked this down to a GC bug: https://bugs.chromium.org/p/chromium/issues/detail?id=847863

@ulan @ofrobots Can this be fixed (and ported to Node 10) soon?

@aalexand I am working on a fix and will make sure that it can be merged back.

@ofrobots do the CLs in crbug.com/847863 fix your repro case?

@ofrobots I also wonder if there is a potential to improve the square complexity mentioned in https://github.com/GoogleCloudPlatform/cloud-profiler-nodejs/issues/199#issuecomment-392493628. It might be not an issue but it probably still burns more CPU in the user's app than it could.

With the CLs in https://crbug.com/847863 I can no longer reproduce the issue. I've requested the merge back on the bug. Once/if approved, @ulan, can you do the merge to the requested branches?
@aalexand I'm +1 on reducing the complexity of the weak callback in the sampling heap profiler. I am not sure about the priority at this point though as it is unlikely to be measurable once the V8 bug is fixed. I'd be happy to guide someone motivated to fix this, but otherwise this would go in our heap profiler backlog.

Once/if approved, @ulan, can you do the merge to the requested branches?
Yep, I'll merge back as soon as I get approval.

This has been merged in V8 6.8 and 6.7. It will get released in the next 10.x release. Too bad it missed 10.4.0 yesterday.

Confirmed by looking at integration tests that we are not seeing this in 10.4.1.


wamp-scram

Continuation of https://github.com/crossbario/autobahn-python/pull/956
This adds an onWelcome/on_welcome hook for authenticators to use and SCRAM uses it to check the server-signature as well as a few other fixups. Now matches the draft-spec (again).

looks good! has the new onWelcome stuff.
rgd the fallout from that: maybe we could add a default (empty) implementation of onWelcome here https://github.com/crossbario/autobahn-python/blob/master/autobahn/wamp/protocol.py#L1129

@meejah I would like to land the current state of WAMP-SCRAM in both AB and CB - do you think it is "ready"? There is some CI fallout that needs fixing, but then I would merge. Could you give me ping when to do a final review for merge? Or just assign me as reviewer when you feel ready ..

Codecov Report

Merging #958 into master will decrease coverage by 0.05%.
The diff coverage is 34.21%.

@@            Coverage Diff             @@
##           master     #958      +/-   ##
==========================================
- Coverage    61.8%   61.74%   -0.06%
==========================================
  Files          71       71
  Lines       12729    12771      +42
  Branches     2030     2034       +4
==========================================
+ Hits         7867     7886      +19
- Misses       4182     4203      +21
- Partials      680      682       +2

Impacted Files
Coverage Δ

autobahn/twisted/component.py
31.69% <ø> (+0.34%)
:arrow_up:

autobahn/wamp/message.py
64.12% <0%> (ø)
:arrow_up:

autobahn/wamp/interfaces.py
100% <100%> (ø)
:arrow_up:

autobahn/wamp/auth.py
49.44% <16.66%> (-0.36%)
:arrow_down:

autobahn/wamp/protocol.py
53.6% <46.87%> (-0.54%)
:arrow_down:

autobahn/asyncio/wamp.py
72.38% <50%> (-0.44%)
:arrow_down:

autobahn/twisted/wamp.py
40.9% <50%> (+0.05%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d4ed72...1d48e90. Read the comment docs.

Okay I added the empty onWelcome and made a few fixups for the tests etc. Should be good to merge @oberstet

perfect! great work. this is a very complete "prototype" I'd say;) now, cool, we have a WAMP-SCRAM spec and client/router prototype=)

@meejah, @oberstet, except for channel binding, is WAMP-SCRAM now fully operational in Crossbar? Does it support dynamic authenticators?

Oh, and thank you for implementing this! :heart:

It doesn't support dynamic authenticators (and no channel-binding, that's correct). Adding dynamic authenticators wouldn't be much more additional work but is not in master right now..

@meejah For internal use, we can do ticket auth with client-side pre-hashing for now. We still have a lot of work to do for user management and permission control. We'd like to "drop in" WAMP-SCRAM later so that we gain the property of an eavesdropper not being able to impersonate a client.


company-files doesn't work after company-tide

What did you expect to happen?
This should work:
(set-company-backend! 'js2-mode
    'company-files 'company-tide 'company-yasnippet)

What actually happened?
Files don't auto-complete unless company-files is 1st on the list, but my company-backends is:
(company-tide company-capf company-yasnippet company-files)

so it says "No completion found" for files.
I tried various hooks like setq-hook and add-hook to put company-files 1st on the list but it doesn't work like something is overriding it.
System information:

emacs   version    26.3
        features   XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD LCMS2
        build      Aug 29, 2019
        buildopts  (--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt' CPPFLAGS=-D_FORTIFY_SOURCE=2 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now)
        windowsys  x
        daemonp    server-running
doom    version    2.0.9
        build      HEAD -> develop cb4f9b86c 2020-03-06 16:37:57 -0500
        dir        ~/.config/doom/
system  type       gnu/linux
        config     x86_64-pc-linux-gnu
        shell      /usr/bin/zsh
        uname      Linux 5.5.8-arch1-1 #1 SMP PREEMPT Fri, 06 Mar 2020 00:57:33 +0000 x86_64
        path       (~/code/web/crwn-clothing/node_modules/.bin/ /usr/local/bin /usr/bin /bin /usr/local/sbin /usr/lib/jvm/default/bin /usr/bin/site_perl /usr/bin/vendor_perl /usr/bin/core_perl ~/.npm-packages/bin ~/.config/composer/vendor/bin /usr/lib/emacs/26.3/x86_64-pc-linux-gnu)
config  envfile    nil
        elc-files  0
        modules    (:completion company ivy :ui doom hl-todo indent-guides modeline nav-flash ophints (popup +all +defaults) treemacs vc-gutter vi-tilde-fringe window-select workspaces :editor (evil +everywhere) file-templates fold multiple-cursors rotate-text snippets :emacs dired electric ibuffer vc :term eshell :checkers syntax spell :tools (eval +overlay) (lookup +docsets) magit :lang cc data emacs-lisp go javascript lua markdown (org +dragndrop +present) php sh web :config (default +bindings +smartparens))
        packages   ((pkgbuild-mode) (package-lint) (livid-mode) (js-doc) (indium) (restclient) (company-restclient) (js-react-redux-yasnippets :recipe (:host github :repo sooqua/js-react-redux-yasnippets :files (*.el snippets))))
        unpin      (n/a)
        elpa       (n/a)

That's how company works. It tries the list of backends, in succession, until one works, then stops there.
If you want to merge the candidates of all those backends, group them:
(after! js2-mode
  (set-company-backend! 'js2-mode
    '(company-files company-tide company-yasnippet)))

Alternatively, use C-x C-f (company-files) and C-x C-s (company-yasnippet) to invoke them directly.
Hope that helps.

That's how company works. It tries the list of backends, in succession, until one works, then stops there.

But then why everything works fine when company-files is 1st on the list, but not the opposite? The code you provide results in this value of company-backends in rjsx-mode:
(company-tide company-capf company-yasnippet
              (company-files :with company-tide company-yasnippet))

Which still doesn't complete file names

How do I remove the company-tide company-capf company-yasnippet from the beginning?

But then why everything works fine when company-files is 1st on the list, but not the opposite? The code you provide results in this value of company-backends in rjsx-mode:

Because company tries each backend in the list until the first one that works, then it stops. company-files worked, so it stops there. If another backend was before it, and it worked instead, then it never gets to company-files.

How do I remove the company-tide company-capf company-yasnippet from the beginning and make my own list for rjsx-mode?

Use this instead:
(setq-hook! 'rjsx-mode-hook
  company-backends '((company-files :with company-tide company-yasnippet)))

(setq-hook! 'rjsx-mode-hook
  company-backends '((company-files :with company-tide company-yasnippet)))

This results in company-backends value being:
 (company-tide company-capf company-yasnippet)

I don't know why.

OK this did the trick:
(when (featurep! :lang javascript)
  ;; Configure autocompletion
  (after! js2-mode
    (defun ~+company-javascript-init-h ()
      (set-company-backend! 'rjsx-mode '(company-files company-tide :with company-yasnippet company-capf)))
    (add-hook 'rjsx-mode-hook '~+company-javascript-init-h)))