Support embargoed dandisets

ATM we are completely skipping them. In the light of integration with github for issue tracking etc would be desired/needed.  In general it should not require much difference in behavior I think, but @jwodder please correct me if I am wrong. I see the need to:

Created github repos would be created private
For all assets populate only API URLs

@yarikoptic would need to look into how we should handle authentication. Likely the simplest way would be to provide (within DataLad) a "provider" configuration to provide the API Token in the request.
I do not think we should populate direct URLs to private bucket ATM at all, in particular since embargo is being redesigned, but we could for "public assets" in embargoed datasets.

If dandiset becomes unembargoed

github repository should be made public if private
we should populate direct http URLs to S3 for all blobs  (worth adding a service command for that: given a dandiset id go through all assets and ensure presence of direct s3 http url)

Some related thoughts

since dropbox backup is "private" anyways, no need to anyhow separate it out for embargoed content

@AlmightyYakob , a few questions:

in the API ATM can we know if blob is embargoed or not -- from URL?
after embargo redesign how would we know that?
do you have an example blob or just a URL to some key configured like designed so that access would be denied -- I wonder if response would provide detail on how to authenticate to gain access. I.e. would it be 403 or 401 with useful WWW-Authenticate

@yarikoptic

Can you provide the ID of a currently-embargoed repository?
In order for the backup program to know when a Dandiset or asset becomes unembargoed, it needs to be able to determine whether a given Dandiset/asset backup was made while the API object was embargoed.  How should that be implemented?

When we create git repo (and subsequent updates), we can set  dandi.dandiset.embargo-status datalad config (so within .datalad/config) to correspond to above embargo_status. When we get to a new update and it is switching from stored EMBARGOED to OPEN [NB1], that is the moment to handle migration.
NB1: since .datalad/config -- we would need to commit it. But we would not know exact time point for when it happened. We should take time of the most recent commit I guess. May be later when audit is implemented (ref: https://github.com/dandi/dandi-archive/issues/525)

@yarikoptic

Should embargoed dandisets be included as submodules in the dandi/dandisets repository?
At what point should support for embargoed assets in partially-embargoed Dandisets be added?  Should it be included in the PR that implements this issue, or should it come later?

For embargoed assets in partially-embargoed Dandisets, should their metadata be included in .dandi/assets.json?

Retrieving embargoed data will require a API key, which backups2datalad previously had no need for.

If no API key is provided explicitly (via an envvar, CLI option, or config option, depending on how we decide to implement passing it), should backups2datalad attempt to retrieve an API key from the keyring?  This would fail if the script is running unattended and the keyring requires a password.
If no API key is provided, should the script error?  Warn?  Silently only operate on non-embargoed data?  Should there be a command-line or config file option to control this behavior?

I do not think we should populate direct URLs to private bucket ATM at all ... but we could for "public assets" in embargoed datasets.

What do you mean by "'public assets' in embargoed datasets"?

Should embargoed dandisets be included as submodules in the dandi/dandisets repository?

good question:
choice A: IMHO it would be ok to include them since we do not leak any metadata which might be sensitive. But it would be a discrepancy from website/API behavior of not showing them up at all unless authenticated/having access.
choice B: For consistency we could

not add them as submodules. To avoid addition/commit by mistake, let's add them to .git/info/exclude.
upon unembargo -- remove from .git/info/exclude and add as submodules

I personally would prefer choice A since it would be less fragile.  @satra WDYT?

At what point should support for embargoed assets in partially-embargoed Dandisets be added?  Should it be included in the PR that implements this issue, or should it come later?

If there is easy way to tell embargoed blob from public -- let's add right away since would be trivial.
If not -- later

For embargoed assets in partially-embargoed Dandisets, should their metadata be included in .dandi/assets.json?

yes. Those repos will be private, so all good here

Retrieving embargoed data will require an API key, which backups2datalad previously had no need for.

yes, will need to be of one admin user. most likely we better gain a dedicated service user, wdyt @dandi/archive-maintainers -- do we have already a dedicated service user / API key?

If no API key is provided explicitly (via an envvar, CLI option, or config option, depending on how we decide to implement passing it), should backups2datalad attempt to retrieve an API key from the keyring?

sounds reasonable

This would fail if the script is running unattended and the keyring requires a password.

how do we address that in any other testing scenarios?  IIRC we do use here a key already to staging for testing

If no API key is provided, should the script error?  Warn?  Silently only operate on non-embargoed data?  Should there be a command-line or config file option to control this behavior?

As embargo is built-in functionality, for now we can just error out.  If later we run into use case to support cases without embargo, we will look into making it optional.

What should backups2datalad do if a Dandiset's embargo_status is "UNEMBARGOING"?  I believe that status means that the Dandiset is currently in the process of being updated from embargoed to open.

Skip update for it and re-consider on the next run

I do not think we should populate direct URLs to private bucket ATM at all ... but we could for "public assets" in embargoed datasets.

What do you mean by "'public assets' in embargoed datasets"?

My assumption was (I think  it might not correspond to how it actually works ATM): it is more of a "public blob" in an asset within embargoed dandiset:  if someone uploads a file which already exists in the archive publicly, we would just reuse blob from the public blobs pool.  Looking at DB models https://github.com/dandi/dandi-archive/blob/HEAD/dandiapi/api/models/asset.py#L137 I was expecting to have a different from blob embargoed_blob in the output of listing assets,

but looking at our "test development" embargoed dandiset where 1 blob also is present in public dandisets -- everything seems to be in embargoed bucket if I look at the details of the asset and the ContentURLs
❯ curl --silent -X 'GET' 'https://api.dandiarchive.org/api/dandisets/000344/versions/draft/assets/' -H 'accept: application/json' -H "Authorization: token $DANDI_API_KEY" | jq .
{
  "count": 4,
  "next": null,
  "previous": null,
  "results": [
    {
      "asset_id": "4ec64802-d979-4917-b7f9-c025c9aeecb1",
      "blob": "bc2f5c33-6022-4249-807b-a6623793d3cf",
      "zarr": null,
      "path": "README.md",
      "size": 121,
      "created": "2022-10-10T19:16:58.431070Z",
      "modified": "2022-10-10T19:16:58.932418Z"
    },
    {
      "asset_id": "8f998a62-ac4c-465d-9989-656c90fb4668",
      "blob": "127efb7b-4261-4f91-80f3-43ba23f3f9f1",
      "zarr": null,
      "path": "sub-private/sub-private.dat",
      "size": 17,
      "created": "2022-10-10T19:16:58.509133Z",
      "modified": "2022-10-10T19:16:58.749121Z"
    },
    {
      "asset_id": "52830095-77e8-4ffe-9635-54ce9cb06567",
      "blob": "2dbaf0fd-5003-4a0a-b4c0-bc8cdbdb3826",
      "zarr": null,
      "path": "sub-public/sub-public.nwb",
      "size": 18792,
      "created": "2022-10-10T19:17:07.870204Z",
      "modified": "2022-10-10T19:17:07.933219Z"
    },
    {
      "asset_id": "684c2229-c4f9-4f4f-91f6-6d55fa1603b9",
      "blob": "10d9c3e5-0caf-4a0b-8662-487e0cba85ce",
      "zarr": null,
      "path": "sub-random/sub-random_size-10M.dat",
      "size": 10485760,
      "created": "2022-10-10T19:19:47.573199Z",
      "modified": "2022-10-10T19:19:48.097682Z"
    }
  ]
}

❯ curl --silent -X 'GET' 'https://api.dandiarchive.org/api/dandisets/000344/versions/draft/assets/4ec64802-d979-4917-b7f9-c025c9aeecb1/' -H 'accept: application/json' -H "Authorization: token $DANDI_API_KEY" | jq .contentUrl
[
  "https://api.dandiarchive.org/api/assets/4ec64802-d979-4917-b7f9-c025c9aeecb1/download/",
  "https://dandiarchive-embargo.s3.amazonaws.com/000344/blobs/bc2/f5c/bc2f5c33-6022-4249-807b-a6623793d3cf"
]

so may be there is no mixed ones ATM. @AlmightyYakob can you clarify on current implementation?

For embargoed assets in partially-embargoed Dandisets, should their metadata be included in .dandi/assets.json?

yes. Those repos will be private, so all good here

I assumed that partially-embargoed Dandisets would have an embargo_status of "OPEN", and thus their repositories would be public.
@AlmightyYakob How do/will partially-embargoed Dandisets (i.e., Dandisets in which not all assets are embargoed) work?  What is the embargo_status of such Dandisets?  How would a user of the API determine whether a given asset was embargoed?

[Fetching from the keyring] would fail if the script is running unattended and the keyring requires a password.

how do we address that in any other testing scenarios?  IIRC we do use here a key already to staging for testing

The backups2datalad testing key is provided solely as an environment variable; the keyring is never consulted.  In addition, dandi-cli tests only interact with the keyring when specifically testing keyring support.

I do not think we should populate direct URLs to private bucket ATM at all ... but we could for "public assets" in embargoed datasets.

What do you mean by "'public assets' in embargoed datasets"?

My assumption was (I think  it might not correspond to how it actually works ATM): it is more of a "public blob" in an asset within embargoed dandiset:  if someone uploads a file which already exists in the archive publicly, we would just reuse blob from the public blobs pool.

How would such assets be identified?

I personally would prefer choice A since it would be less fragile. @satra WDYT?

if it's just a dandiset id, that's fine. if it's the tree layout, we shouldn't. we don't enforce that embargoed dandisets should be valid or anonymized explicitly at the moment.
@jwodder - currently there is no support for a partially embargoed dandiset (it's either embargoed or not). this is a feature that has been requested, which if implemented would result in multiple status at the dandiset level and asset level protection. we can change the backup actions when partially embargoed or restricted dandisets are implemented.
as @yarikoptic noted a public blob can be added to an embargoed dandiset, but that blob should remain in the public bucket till modified.

@satra

I personally would prefer choice A since it would be less fragile. @satra WDYT?

if it's just a dandiset id, that's fine. if it's the tree layout, we shouldn't. we don't enforce that embargoed dandisets should be valid or anonymized explicitly at the moment.

yeap -- just dandiset id, and possible its datalad dataset ID. Ok, then let's proceed with the simpler A.

as @yarikoptic noted a public blob can be added to an embargoed dandiset, but that blob should remain in the public bucket till modified.

that is what I thought but not sure if that "can" is really implemented -- according to collapsed demo in my comment above even that publicly available blob is said to be available from embargoed bucket, not the main one.
I will let @AlmightyYakob to clarify on @jwodder's questions.

@AlmightyYakob How do/will partially-embargoed Dandisets (i.e., Dandisets in which not all assets are embargoed) work? What is the embargo_status of such Dandisets?

The questions you're asking are a great example of why an embargo re-design is needed. At the moment, there is very little restriction on what is allowed. Indeed dandisets can have mixed embargoed/open assets, and the dandiset containing these mixed assets could be either embargoed or open.

How would a user of the API determine whether a given asset was embargoed?

The only way you could really do so is by looking at the access metadata field on the asset itself, which should be either dandi:OpenAccess or dandi:EmbargoedAccess. However, due to the faults of our current implementation, that can't be relied on. Just messing around to answer this question, I exclusively found example of embargoed asset blobs that pointed to an asset with an access field of dandi:OpenAccess.

@yarikoptic Taking @AlmightyYakob's comment above into account, how should assets with different embargo states from their Dandisets be handled?

ok, for now let's not discriminate between different kinds of assets -- and for an embargoed dandiset populate only with API urls.

For reference, a summary of what needs to be done for this issue:

backups2datalad should now always require an API key to be provided via the methods supported by DandiAPIClient::dandi_authenticate()
Retrieval of all Dandisets from the API should now set embargoed=true (Due to https://github.com/dandi/dandi-archive/pull/1790)
If a Dandiset has an embargo_status of UNEMBARGOING, skip operating on it for the current run.
If a Dandiset has an embargo_status of EMBARGOED:

Set dandi.dandiset.embargo-status in .dandi/config to EMBARGOED
The GitHub repository for the Dandiset and any Zarrs within it should be private upon creation
Only register API URLs for assets & Zarr entries, not S3 URLs
The dataset should still be added as a submodule to dandi/dandisets

If a Dandiset has an embargo_status of OPEN and dandi.dandiset.embargo-status in .dandi/config is EMBARGOED:

Register S3 URLs for all assets and Zarr entries

Add a service command for doing just this

Set dandi.dandiset.embargo-status in .dandi/config to OPEN

When committing this change (if there were no other changes?), set the commit's author date to the same value as for the previous commit to the Dandiset

Make the GitHub repository for the Dandiset and any Zarrs public

(Yes, I'm aware that embargoed Dandisets currently can't contain Zarrs, but extending the logic to Zarrs right away shouldn't be that hard and is a good idea.)
@yarikoptic Is this all correct?
Side note: I'm assuming that an open Dandiset cannot be converted to (fully) embargoed.  Is that currently correct?  Will it remain correct after the redesign?  CC @jjnesbitt

@yarikoptic I've created a separate issue for supporting partiall-embargoed Dandisets later at #10.

Overall looks great, thank you. A few clarifications:

Set dandi.dandiset.embargo-status in .dandi/config to EMBARGOED

in .datalad/config not .dandi/config

Only register API URLs for assets & Zarr entries, not S3 URLs

As embargo support is still being worked out and changing, and there is even no design for zarr embargo and I don't think we even have API to provide access to embargoed zarrs entries --  just error out if encountering zarr entry in embargoed dataset as should not happen ATM.  That would also simplify implementation as would not need recurse into zarrs to populate URLs and making zarrs public in the last item.

@yarikoptic How exactly do you want the API key to be provided when running the backup script?  As an environment variable, on the command line, in the config file, via the keyring, what?  Note the following:

backups2datalad uses a custom async client rather than DandiAPIClient, so it does not have the latter's keyring authentication methods
Currently, keyrings are not involved at all while testing backups2datalad
Reliance on keyrings could be a problem when backups2datalad is run unattended if the keyring happens to need a password

dandi-cli's keyring code also requires user interaction if the keyring lacks an API key, if the key is invalid, or if the default keyring fails

I am ok with just relying on os.environ.get("DANDI_API_KEY", None) for now.

May be for clarity and consistency on a separate commitment would be better. Might be even worth tagging it (eg unemargoed, just lightweight not annotated) to simplify introspection etc

@yarikoptic What git-annex command do I use to query whether a key exists in the repository's HEAD?

One key at a time? What do we need it for?

@yarikoptic

One key at a time?

Preferrably batched, if possible.

What do we need it for?

When a Dandiset is unembargoed, backups2datalad needs to register the S3 URL for each asset with the corresponding entry in the backup repository, and git-annex registerurl operates on keys.  If the code were to try to register a URL on a key that's not in the repository — say, if an asset were added to the Dandiset in the Archive since the last time backups2datalad ran — registerurl would fail, so we need to check whether an asset's key is in the repository before trying to register a URL for it.  Note that trying to implement this by checking whether the asset path exists wouldn't work in situations where an asset's blob was modified since the last time backups2datalad ran, as then we'd be registering the URL for the new blob on a file referring to the old blob.

so, overall, we need to get through all annexed files in the HEAD tree and registerurl them, correct?
if datalad api was used, then smth like datalad -f json_pp status --annex basic (ds.status(annex='basic') in python I assume) would have returned records for all the files with the annexed ones having keys listed.   Underneath it uses find so equivalent command would git annex find --format='${key}\n' --include '*' to list the keys, just take a unique set of them since I guess would repeat for duplicated content.

@yarikoptic

so, overall, we need to get through all annexed files in the HEAD tree and registerurl them, correct?

Correct.
Moving on to a related aspect: Checking keys requires getting assets' SHA256 digests from the Archive.  What should the "S3 URL registration routine" do if the Archive hasn't finished digesting an asset?  For comparison, the normal asset-syncing code (which, for various reasons, the new routine is separate from) logs a message when this happens and doesn't download any further assets.

@yarikoptic Ping.

What should the "S3 URL registration routine" do if the Archive hasn't finished digesting an asset? For comparison, the normal asset-syncing code (which, for various reasons, the new routine is separate from) logs a message when this happens and doesn't download any further assets.

sounds like keeping the same behavior would be logical here as well, WDYT?


Ramped movement and squared rotation

Ramped normal input and squared rotation input should be implemented in the WheelSystem class.

It might be nice to also have ramping for sideways input.

@KIllin-A13, right now only forward ramping is unit tested. We need to unit test sideways ramping too.

Let's leave this open until sideways ramping is unit tested.


[Sema][WIP] Redundant Conformance Diagnostic should have an associated fixit

PR basically adds fixits to redundant conformance diagnostics. @CodaFi helped me getting into this bug. The idea was to add SourceRange to ConformanceDiagnostic that will let to emit fixit.
As I understand I need to get SourceLoc of previous token (":" or ",") and compose SourceRange starting ":" or "," ending ConformanceDiagnostic.Loc. But I don't know how to get SourceLoc of previous token. Help wanted.
// Test cases:
protocol A {}
protocol B {}
protocol C {}
struct Foo {}

extension Foo: A {}
extension Foo: A {} // after fixit -> extension Foo {}
extension Foo: B {}
extension Foo: C, B {} // after fixit -> extension Foo: C {}

Resolves SR-5058.

So sorry for not reviewing this earlier.

It seems like somebody already got to this in 1214524.  However, the point about the fixit still stands.  If you'd like to take that on, it should be a simple change to make.
Again, I'm so so sorry for letting this sit for a year. I'm trying to make an effort to respond to all the old pull requests, and implement stuff to make sure things don't fall by the wayside again.


Artakm/bootstrap v3 update

Updated IdentityUI V3 to reference the 3.4.0 version of Bootstrap

The reported test failures are from unrelated area, so I'm moving ahead with merging this change in.


Extensions to the Paper

Hello,
Thank you providing this useful repo, I enjoyed reading your paper and the work is really interesting. I had a few questions about extensions to this work that I could not find the answers to in the paper, if you have any insight that would be greatly appreciated:

How would this method extend to more than 2 modalities? The discrepancy ratio equations are set up for a binary situation, could the ratio be calculated for 3 or more modalities?

Would this method work for multi-task output? Say the network outputs a binary cat/dog output, and a separate black/brown/white fur output. How would on calculate the scores then in eqs. 8 of the paper, as each input would have a multi-label output.

I am doing training with individual learning rate scaling of the params in each modality's encoder (initial lr of visual encoder is 0.1, initial lr of audio encoder is 0.01 for example), how would the calculation of the coefficients in eq. 10 of the paper be changed? Or should this not factor into the modulation?

Thanks again for any information you may have.

Hi, thanks for your questions.

 I consider that one simple way is that calculated the one-to-others discrepancy ratios pair-wise, then average the results.

Also, we have not attempted the multi-task scenarios yet. The multi-task scenarios are more complex since the balance among losses should also be taken into consideration.

The learning rate and batch size are two factors affecting SGD noise. Hence, when the value of the learning rate of two modalities are different, the gradient modulation degree is ( eta*k, eta is the learning rate, equ 11).


Upgrade python version for unittest on github

Error running unittest on github due to the following error
from typing import final
ImportError: cannot import name 'final' from 'typing' (/usr/share/miniconda/lib/python3.7/typing.py)

Depends on https://github.com/facebookresearch/d2go/pull/59


Random hangup episodes in Campaigns production

BUG
Current Behavior

Lately, we have been having random episodes of timeouts in our production app on Heroku.
These hangup periods are resulting in 503 errors and messages being accumulated in out deadLetterQueue.

Desired Behavior

This random hangups should not occur.

Steps to Replicate

Randomly we still don't know what triggers it.

Why This Matters

It's breaking Gambit by leaving it in an unpredictable "hanging" state.

Last random hangup metrics graph.

Just an update: As of Nov 8th. These hangups have not occurred again.

Update: As of Nov 29th. These hangups have not occurred again. Closing this issue.


MPI-related fixes and improvements.

This PR does not fix #81, but it does implement the following fixes:

Make *(::AbstractHamiltonian, ::MPIData) (also three-argument dot) type-stable and faster.
Change hashing function in targetrank for better distribution among ranks.
Fix dot product bug in AllOverlaps.
Add mpi_allprintln for nicer printing with MPI.

Pull Request Test Coverage Report for Build 927100240

30 of 44 (68.18%)  changed or added relevant lines in 5 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.2%) to 80.464%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/RMPI/helpers.jl
1
7
14.29%

src/RMPI/mpidata.jl
22
30
73.33%

Totals

Change from base Build 916950206:
-0.2%

Covered Lines:
2356

Relevant Lines:
2928

💛  - Coveralls


Define some first-line and second-line support processes

Background and proposal
There are often cases where our support process is under-documented. For example, a few questions that people weren't sure how to answer:

How should we prioritize support requests?
What should we do if a request is not immediately "closeable"? Or if it requires ongoing follow-up work?
How can we communicate our inability to fulfill a request?
What kind of communication should we use throughout the support process?

We should document some rough guidelines for these common questions, and also provide references to documentation about how to carry out first-line support.
Implementation guide and constraints
Another way to approach this is to ask "what are some common support situations, and what should we do in each situation?" We can draw from our experiences thus far to agree on some team practices to follow.
Updates and ongoing work
No response

cc @sgibson91 who I believe has some notes about support processes at other organizations that we could use for comparison.

Notes on Managed Service Provider (MSP) first-line support protocols
I had a chat with my partner who has done first-line support in a couple of companies and this are my notes from that discussion.

Note: This will be from an industry perspective and will not be entirely applicable to us. We should pick and choose what we think would work for us.

Responding to tickets

We should setup Freshdesk to send an auto-response to all incoming tickets saying we have received the ticket and issuing a reference number (if it does not already do this?) (Sarah is not sure what emails get sent to clients when certain buttons are clicked and this fuels some of her anxiety.)
First response from the Support Steward should be within a given timeframe of the ticket coming in, corrected for active support hours (i.e., what timezone is the person providing support in vs. the support requester). An entirely unrealistic for us example might be: First response from Support Steward should be within half an hour of the ticket coming in.
Escalating to third parties (e.g., GCP or Azure...)

Support Steward would provide a message to the client indicating they've escalated the ticket to a third party and are waiting on a response
Ticket state is "On Hold"

Prioritisation of tickets

A priority queue with all integers set to maximum is just a queue...

P5: Service request - This is an "I would like..."-style ticket
P4: "I have a problem and it affects only me"
P3: More than 5 people or a whole department can't do the same thing.* (Or request comes from a VIP)
P2: Issue that affects multiple departments but not the whole company. Perhaps more than one service is down.
P1: No one can work. Everything is on fire.

Note: If there is a temporary work-around for a P3 problem, you can apply this fix and downgrade the ticket to a P4 while working on the real fix. Often this is when tickets get pulled out into projects with proper project management as this can affect SLAs agreed with the client. This project work is then billed back to the client.

SLAs regarding timeframes for carrying out support-related tasks
These are the timeframes we agree with a client for resolving problems. They should be defined in number of working hours within our active support hours bracket (I'm cheating in the example below).

These are another set of example timeframes that would be unrealistic for us to adopt

P5: Resolve within 5 working days unless it needs to be a project
P4: Resolve within 3 working days
P3: Resolve within 1 working day
P2: Maximum 4 working hours to resolution
P1: ~1-2 hours to resolution
Communications during P2/P1 events
P2: Provide updates every 2 hours
P1: Provide updates every hour
Have a comms template for these events. From what was described to me, this is very similar to the top half of our Hub Incident issue template. Includes a timeline of the event, the symptom reported, etc. These comms should still go out regularly, even if the update is still "we're investigating".
What channels do these comms go out on? Mailing lists, forum posts, twitter?
After the event is resolved, compile and release an autopsy report. This is very similar to the second half of our Hub Incident template. Covers what went wrong, how did it go wrong, what are the steps to prevent it happening again or minimise disruption if prevention isn't possible.
Support Tiers

Pay as you Go
Tier 1

"break & fix" (something is broken, we fix it)
Charge £X per Y hours per month

Tier 2

break & fix
provide some recommendations for improvement
Charge £X+dX per Y+dY hours per month

Tier 3

Unlimited support hours per month
Extra services such as 24/7 monitoring, automatic upgrades

This is potentially something we could fit into our alpha service pricing matrix
Guidelines for what is in scope of our support model
We need to define these (maybe on a per client basis?) and be strict about it.
The "maximum capitalist company" thing to do might be:

If a problem is found to be a client's fault

We offer guidance to help them fix it
If they really want us to do the work, fine but we have a flat hourly rate of £X

I'm not 100% convinced that's the right thing for 2i2c to do, just presenting it here as one aspect I learned about.
Disadvantage
(And a pretty big on IMO) This style of MSP support requires time-tracking in order to bill the client for, e.g., overtime on support hours, projects generated from support tickets, etc.

Some info from a friend at a tech company
I also had a conversation with a friend at a tech company about this that runs internal and external software services (not mentioning company or friend's name for privacy purposes). Here's a short breakdown of their process:
Roles they use

An engineering team broadly understands the infrastructure behind the service. They do a combination of development and operations.
An operations role is a member of this team that will attempt to resolve all operational issues first. This rotates weekly through the team.
A support person is a dedicated role that is always held by the same (non-engineer) person. This person communicates with the customers and forward information to the Operations Role on the engineering team for resolution.
A project manager largely oversees timelines and planning around development efforts, but may decide to change deadlines if enough support issues pop up that the team doesn't have time to meet them.
A team manager is more like a "line manager", they help with the team processes but focus more of their efforts on making sure team members are supported, on a good career path, etc.

When a support request comes in
Here's what happens:

The support person interacts with the person that made the request - they respond and try to identify what's going on.
If an action needs to be taken, they contact the operations member for that week, who tries to resolve the issue.
If they cannot resolve the issue, they speak to the Project Manager, who identifies another member of the team to help with the issue.
If they cannot find another person to assign to the issue, it goes into a list of open operations issues with no owner.
Each week, they have a triage meeting led by the support person. The outcome of that meeting is that every operations / support issue must have an owner.
In this meeting, the Project Manager may help estimate capacity of team members and suggests the person to take on an issue, if none of the engineers want to pick up the issue. Apparently there are often "awkward silences" where they wait for somebody to volunteer to do something 😆

How this interfaces with development
They do development in parallel with these operations tasks, and each team member has ongoing projects with deadlines associated with them.
Occasionally, there are enough operational tasks that they realize they won't hit their deadlines. When this happens, the Project and Team Managers discuss with one another and agree on a plan forward, potentially to move back the deadlines for their projects.

Thank you both for sharing these pieces of information!
Everything described here pretty much aligns with my own previous experiences!
I think several of the pieces detailed above could be adopted with adjustments accordingly to our current state and our mission/vision.

Update
@yuvipanda had some great ideas in https://github.com/2i2c-org/infrastructure/issues/1154 for steps in this direction, along with process notes shared from Sarah. I think we should make a quick push to document some of that, since the content is mostly there. We could also use this as an opportunity to update our SLA docs a little bit to make them more clear.

I'm going to close this one and say it was completed by merging the following PR:

https://github.com/2i2c-org/team-compass/pull/422

We can continue to iterate on these team processes over time!


Enable sourcelink and deterministic builds

I would recommend this. See small examples here:

https://github.com/NLog/NLog.Extensions.Logging/pull/439
https://github.com/NLog/NLog.Extensions.Logging/pull/479

@304NotModified thanks for the suggestion.
I've never used Sourcelink before, so I'd appreciate it if you can please review the associated commit?
As far as I checked it's working - because in the generated nuspec I can see <repository type="git" url="etc" commit="etc" />. but just want to double check.

I dont see anything strange,  but I recommended to check the nuget package with nuget package explorer.
See https://github.com/NLog/NLog.Extensions.Logging/pull/479#issuecomment-808529410

Thanks for the tool suggestion. Turns out that both Source Link and Deterministic Builds were not working :-(
I could fix Source Link by adding the PDBs into the nupkg (yeah, I know that the current recommendation is to leave pdbs in a snupkg and upload it into a symbol server, but making it all together seems to be easier).
Took me lots of attempts to make the deterministic build work. Looks like using dotnet pack will randomly work (fails most of the time), while with MSBuild it works most of the time (or always). So basically I started using MSBuild to pack (like you do in NLog), which works better.
Thanks again for the advice and for the links!

I could fix Source Link by adding the PDBs into the nupkg (yeah, I know that the current recommendation is to leave pdbs in a snupkg and upload it into a symbol server, but making it all together seems to be easier).

Creating a snupkg is pretty easy:

/p:SymbolPackageFormat=snupkg (msbuild)
Upload also the snupkg to nuget.org

That's because the parameter belongs to the build phase and not the pack, I guess

As far as I understood the dotnet pack also rebuilds (unless we specify --no-build which would get the existing build), and therefore the flag should work in both dotnet build and dotnet pack.
So my wild guess is that dotnet tool is not correctly passing the flags or something (I've seen some other threads reporting issues with dotnet but none having issues with msbuild).

looks great!

(checked also the other 2)


Make local builds

What this PR does / why we need it:
Add a few targets to make it possible to build code and test binary and also run functional tests locally.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:

NONE

ok, the failed test is a CDI - KVP coordination bug,
/override

/override pull-kvp-functional-test

/lgtm

/approve


adding since_last_completion_seconds metrics for basebackups/walg_last_upload

Hey,
I'm just started to use zalando pg operator and this exporter works really nice, thanks! 👍
Would it be possible to add new since_last_completion_seconds metrics for (oldest|newest)_basebackup|walg_last_upload, this will help to monitor these backups with SaaS like datadog, which doesn't have time() function in monitors 😒

ping @thedatabaseme maybe for better visibilty


'light' theme needs some love

🐛 Bug
The 'light' theme for napari is looking pretty terrible these days:

We should update it and make sure that any PRs that touch the UI are tested with both themes.
Also worth noting that the base layer needs to have blending mode "translucent" — with additive, you get a totally white canvas. Perhaps it's a chance to play with inverted LUTs or something, not sure.
Finally, I'm not sure whether there is a way to set the theme globally. I could set it on a Viewer instance, but I could not set it on the napari.Viewer class. It seems to me that it should be more of a global than an instance setting.
To Reproduce
Steps to reproduce the behavior:
import napari
v = napari.Viewer()
v.theme = 'light'

This crashes:
import napari
napari.Viewer.theme = 'light'
v = napari.Viewer()

This makes sense because it replaces a property with a text string! But I couldn't find how else to set the theme.

Bonus issue: changing back to dark doesn't work fully:
v.theme = 'dark'

good to flag!! I'll take a look. our stylesheets need some love in general too

Any reason not to deprecate theme options, and make dark the only option? Having theme options makes UI iteration much more expensive.

@jni theme looks fine on my mac so this might be something cross platform :-(

@marciovm we've had comments from people who prefer light to dark. The way we've set it up should make it pretty easy to maintain both - we have color palettes and style using the palettes, but something funky seems to be happening above :-(

@sofroniewn maybe not right place to discuss but isn't it better to keep canvas background black on both themes?

Discussing here is good @AhmetCanSolak. I'm completely open to that, and it's easy to change, but curious why do you think black is a better canvas background for both themes? I guess it might be more consistent, but I though it looked nicer to have a white background with the light theme. @jni were you suggesting this too? I think @alexdesiqueira was someone who was excited about the light theme if I remember correctly :-) so curious what they prefer

I think black background is more comfortable as white could possibly interfere with image edges or with some other layers. Not a big deal but just to me black background by default makes more sense.

@sofroniewn please note the problem above. If someone uses code where all the layers have blending mode "additive" (as channel_axis=n does), then the image is just all white if the canvas is white. That's a problem.
Regarding the platform dependence: I'm on Catalina, are you? And, can you try with a fresh install from pip? (I am trying it now, but am on a crappy connection so it might take me a while.)

Hey all,
thanks for remembering @sofroniewn! I don't care about the default, if there's an option for a light theme. :blush: however, @marciovm put a good point: how expensive it is to maintain other themes?

reverting to black as a default background makes sense for the reason @jni provides. we can do that soon.
I am on mojave:

I'll try again on my old MBP when I get home and report back. If any other Mac users want to report their results here, we would be grateful!

mine looks more like @jni   I'm stuck on high sierra cause I want to use my NVIDIA card and apple sucks.  (it's a hackintosh for what it's worth... but I wouldn't expect that to affect this).

napari, 0.2.4+12.g6c5bb037
Qt, 5.13.1
NumPy, 1.17.3
SciPy, 1.3.1
VisPy, 0.6.2
scikit-image, 0.16.1

Platform: Darwin-17.7.0-x86_64-i386-64bit
Python:   3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 14:38:56)
   [Clang 4.0.1 (tags/RELEASE_401/final)]
Backend:  PySide2
GL version:  '2.1 NVIDIA-10.33.0 3<IP_ADDRESS>.15.108'
MAX_TEXTURE_SIZE: 16384

here's a different mac, this one a macbook pro on mojave.  Interestingly, this one didn't even try in the controls area.  Strange that we're seeing more than just 2 phenotypes!

napari, 0.2.4+12.g6c5bb037
Qt, 5.13.1
NumPy, 1.17.3
SciPy, 1.3.1
VisPy, 0.6.1
scikit-image, 0.16.2

Platform: Darwin-18.7.0-x86_64-i386-64bit
Python:   3.7.3 | packaged by conda-forge |
   (default, Jul  1 2019, 14:38:56)  [Clang 4.0.1 (tags/RELEASE_401/final)]
Backend:  PySide2
GL version:  '2.1 NVIDIA-12.0.24 3<IP_ADDRESS>.10.103'
MAX_TEXTURE_SIZE: 16384

looks like I'm on PyQt5 by default - lol!
napari, 0.2.5+1.gbc92411
Qt, 5.12.3
NumPy, 1.17.2
SciPy, 1.2.1
VisPy, 0.6.3.dev7+g839ada88
scikit-image, 0.15.0

Platform: Darwin-18.5.0-x86_64-i386-64bit
Python:   3.7.3 (default, Mar 27 2019, 16:54:48)  [Clang 4.0.1 (tags/RELEASE_401/final)]
Backend:  PyQt5
GL version:  '2.1 INTEL-12.8.38'
MAX_TEXTURE_SIZE: 16384

so that maybe it ...

Nope! It's not!!! No change for
Platform: Darwin-18.5.0-x86_64-i386-64bit
Python:   3.7.3 (default, Mar 27 2019, 16:54:48)  [Clang 4.0.1 (tags/RELEASE_401/final)]
Backend:  PySide2
GL version:  '2.1 INTEL-12.8.38'
MAX_TEXTURE_SIZE: 16384

still looks great on mine

couldn't be Qt 5.13 v 5.12?

maybe it's Qt 5.13, I can try upgrading.
Also @jni @tlambert03 - if you overwrite the hardcoded default to be light here https://github.com/napari/napari/blob/6c5bb037e191bce3c05c5bf23f4f143ecc89a240/napari/components/viewer_model.py#L87
how does it look? i.e. is this about switching, or will it still look bad?

upgraded my PySide2, but no change - bellow still works for me.
napari, 0.2.5+1.gbc92411
Qt, 5.13.2
NumPy, 1.17.2
SciPy, 1.2.1
VisPy, 0.6.3.dev7+g839ada88
scikit-image, 0.15.0

We should also print the PySide2 / PyQt5 version # in that top part too

yeah, it's the switching.  If I hard-code in the theme it looks fine, and here's what it looks like when I switch back (inverted problem from above).  so that should help narrow it down

Switching in Qt has some known problems - we often need to call these commands polish and unpolish to get around them, I imaging throwing a couple more of them in there will fix this, what is so strange to me is the difference between my machine and yours!!!

In general our stylesheets are a bit of a mess right now (which is likely independent of this problem), but we should probably give organizing them some attention, it might also speed up the load time (though I don't know that)

@sofroniewn perhaps whatever difference was causing your double title bar (which only happened on your machine iirc) is also causing this??? LOL
have you tried a pip install in a clean environment? I wonder if you have some artefacts in your working directory that somehow makes this behave nicely.


GEDI_L3 fixes

EPIC Link
https://github.com/MAAP-Project/maap-documentation/issues/223
Acceptance Criteria (Tasks):

[x] fix broken links
[x] error: "np" not defined in last cell

made changes, moving onto VEDA guidelines ticket since changes were very minor


travis: Run backend tests in Python 3.5.

@timabbott, done.

Great -- thanks for working through this!
Would you add a line to make tools/travis/backend print out the Python version it's using (with python -V) near the beginning? That'd just help us reassure ourselves whenever working on this in the future that it really is all wired up right and isn't, say, secretly running the system Python 3.4 instead.

@gnprice, done. I have also added some explanation in the commit and comments.

Merged!  Thanks @umairwaheed!

Thanks.


sponsor / paypal / patreon

@porres if you have a PayPal account, please log in and set http://paypal.me to work - as that would help - you might be able to get something like https://www.paypal.me/alexandertorresporres and point people to that (paypal me porres is already taken)


qt6: Handle deprecated QLocale::nativeCountryName

Split from https://github.com/bitcoin/bitcoin/pull/30997.
QLocale::nativeCountryName() has been deprecated since Qt 6.6.
QLocale::nativeTerritoryName() was introduced in Qt 6.2.
This PR ensures compatibility across all supported Qt versions.
No behaviour change for the current codebase, which uses Qt 5.

This PR can be tested on macOS using the first commit from https://github.com/bitcoin/bitcoin/pull/30997 and Homebrew's qt package.

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews
See the guideline for information on the review process.
A summary of reviews will appear here.

Friendly ping @Sjors @furszy @achow101 @jarolrod ;)


Delete session async bugfix

Fixes a bug where the delete session async failed to delete sessions due to missing credentials.

@danielgsims could you please resolve the conflict?


event when dragging started

Not clear from doc how to add handler to start dragging event
for example to highlight dropzone.
I've tried with no success
$(".draggable").draggable()
  .on('dragging:start', function(){
    console.log('dragging:start')
  });

dragging:start is an internal event. You could use
.droppable().on('droppable:activate', function() {})

or use the activeClass option of the droppable:
.droppable({ activeClass: 'drop-here' })

Does this work for you?

Thank you for fast answer.
droppable:activate is works perfectly.
Could you please help with some other issue,
drag proxy during dragging has bad z-index and goes under other elements.
Is any way to define drag-proxy styles?

I'll add an option for this

.draggable({ cloneClass: 'foreground' })

Should work now (using master)

It looks like cloneClass doen't work from current master

I've added cloneClass to the docs: http://ma.rkusa.st/touch-dnd/draggable.html
Maybe, the example does work for you? If not, which browser are you using?

I've also added draggable:start and draggable:stop events

Please feel free to reopen, if the issue still exists


README pictures expired

As is shown below:

The pictures in README do not show properly no matter using proxy or not while this problem not occurring in other GitHub sites. Probably the links of these pictures are invalid. Consider update them.

In addition, images "设置仓库" "环境变量" "二级域名" "一键部署" "部署过程" "唤醒云引擎" "mail-blue-template" "彩虹模板" "重启云引擎" now showing, but images "评论管理" and "创建定时器" can be shown correctly.

Thx. Readme updated now.


No guidance for exporting / importing docker volumes

File: storage/volumes.md, CC @gbarr01
There is no guidance on this page for exporting and importing docker volumes.
The 'docker export' command reference contains a link to a non-existent section on this page 'https://docs.docker.com/engine/tutorials/dockervolumes/#backup-restore-or-migrate-data-volumes', implying there used to be some documentation for this and it has been removed.
Moving a volume from one container host to another seems like it should be a straightforward task, as it is for containers with 'docker export' and 'docker import'.

Closing this ticket due to its age, and the impending refactor. If you think this is in error, feel free to reopen. Thanks!

I'd like to re-open - there's still no documentation on how to export a volume from one machine and import it onto another.

Reopened in https://github.com/docker/docker.github.io/issues/7963


feat: Add support for target and rel attribute for hero action links

What kind of changes does this PR include?

Changes to Starlight code

Description
This pull request changes the CallToAction component so that the target and rel attribute can be set.

I will create a new clean pull request.


[Find] Fix #14094 az find Fix Queries failing when not logged in and when telemetry is disabled

Description
The Aladdin service expects to receive a GUID for both the correlationId and subscriptionId if they are sent. If users were not logged in or had disabled their telemetry, this service would return a 400, since the values passed in were considered strings, not GUIDs. The error message from the server is no longer displayed. It is now a more user friendly message.
I believe #14094 was caused by this issue, though the user was unable to reproduce it, so there could be other factors at play.
Testing Guide
az find "create a vm"

This checklist is used to make sure that common guidelines for a pull request are followed.

[X] The PR title and description has followed the guideline in Submitting Pull Requests.

[X] I adhere to the Command Guidelines.

hi @bim-msft @fengzhou-msft  could you pls help to review this PR? thanks.

Looks like the style check in the CI/CD is failing, but not because of az find, assuming I'm reading it correctly. Would re-running it again fix the issue?

Looks like the style check in the CI/CD is failing, but not because of az find, assuming I'm reading it correctly. Would re-running it again fix the issue?

Style check passed after re-running, but CI status is not updated in the PR. It needs force merge. @yungezz


Add maintainer for protobuf-specs

William has significantly contributed to the Python client library and has been active in discussions and reviews around client specifications and this protobuf library.
Also removed one inactive admin (they're still a maintainer) and cleaned up duplicated maintainer (specified in users.yaml).

Summary

Release Note

Documentation

cc other maintainers for +1s: @kommendorkapten @znewman01
cc for approval: @bobcallaway @priyawadhwa @trevrosen

(@kommendorkapten fyi you're still a maintainer, just removed from collaborator since you're already in the team with maintainer perms)

@haydentherapper please rebase we did some changes to add another org :)

Done


proposal: runtime: flag to disable runtime override of GODEBUG

I would like to make a change to the go runtime to convert the debug variables parsed from GODEBUG to constants, and disable GODEBUG overriding at runtime.
Many of these constants are used to gate low level checks in the depths of the garbage collector or runtime, and if they were true constants, some of these checks could be elided by constant propagation. An example is the findobject function which would be able to skip some checks if debug.invalidptr were known to be a fixed constant 0. (internal reference b/288410395, these have substantial cost)
I would like to propose a new flag to cgo, strawman fixed_godebug, that when provided generates constant values for these fields as-if the GODEBUG env var had the value of fixed_godebug at runtime.
The work needed to implement this would be:

Split out debug itself and functions that modify debug.* fields into a new file, runtime/godebug.go
Move all debug fields to top level variables named debug* and all their usages in the runtime/ package. These are not exported so should not be breaking.
When the fixed_godebug flag is not empty, have cgo generate a replacement for godebug.go that replaces the var block with debug* values with a const block that sets all values inline, and makes all modifying functions log and do nothing.

Why add it to the cgo tool?

Why add it to the cgo tool?

I'm not super picky about where its added, is there a better place you would suggest to perform this manipulation?

I don't understand what cgo has to do with this at all.
More importantly, you said what you want, but you didn't say why you want it.  What is the goal here?  What is the problem being solved?  Thanks.

I don't understand what cgo has to do with this at all.

some piece of tooling needs to swap out the godebug.go file inside of the go runtime that has:
var (
debugcgocheck           int32
debugclobberfree        int32
debugdontfreezetheworld int32
...
)

to:
const (
debugcgocheck           int32  = 0
debugclobberfree        int32 = 0
debugdontfreezetheworld int32 = 1
...
)

Is there a better piece of the compiler ecosystem to do this?

What is the goal here? What is the problem being solved? Thanks.

Based on horizontal profiling, a lot of cycles are spent within the go garbage collector (and presumably in other places in the core runtime) on performing boolean checks that would be removed via constant propagation if the debug variables were constants. I would like to allow users (like myself) who don't want these debug checks to reduce the overhead of the go runtime further by disabling these constants at compile time instead of runtime.

Is there a better piece of the compiler ecosystem to do this?

Maybe there is some terminology issue here.  In the Go ecosystem cgo refers to https://pkg.go.dev/cmd/cgo.  That tool is not going to be swapping out any files in the Go runtime.  That's not what it's for.  But this is a detail.

Based on horizontal profiling, a lot of cycles are spent within the go garbage collector (and presumably in other places in the core runtime) on performing boolean checks that would be removed via constant propagation if the debug variables were constants.

If that is the only goal here, then our first step should be to try to analyze this problem and see if we can fix it in the compiler or runtime in some way, without having to change the semantics of GODEBUG.  Would you like to open a different issue with some profiles showing the problem?  Thanks.

Maybe there is some terminology issue here

Yes, this is my fault. I think the tool I should have referenced was gc, but my intent was for this to be built into the compiler infrastructure itself, whichever piece makes the most sense.

then our first step should be to try to analyze this problem and see if we can fix it in the compiler or runtime in some way

The problem trying to be fixed is: there exist runtime branches and checks in compiled go code (in the garbage collector in particular) that incur a lot of cost, but only exist because GODEBUG can be changed at runtime. If GODEBUG is fixed at compile time, these branches and checks go away.

Generally these debug variables are (carefully) not checked on hot paths, or are placed behind a (predictable) check we have to do on the hot path anyway. For invalidptr checked in findObject specifically, that falls under a path that's taken basically only for (basically) broken programs. In real-world use that branch should never be taken. There may be something to save there in terms of instruction cache pressure? That's an interesting one too since I think it only actually does anything if the compiler builds with -clobberdead as well. That seems like it might just be cleanup (absent any performance difference even) to make it compile-time? CC @cherrymui
If I'm wrong, that's great, and that's nice low-hanging fruit, but I'd like to see some more evidence that these debug variables are actually impacting performance. It would be pretty straightforward to patch out certain branches and run some of our benchmarks just to get an idea of the difference.
Also, I think the right thing to do for at least some of these, if they were to become compile-time constants, would be to turn them into GOEXPERIMENTs, kind of like how we did for GOEXPERIMENT=cgocheck2, which used to be GODEBUG=cgocheck=2 but had to change for other reasons.


feat: Add Rust toolchain to Hubble and move ed25519 to rust

Motivation
Doing CPU intensive cryptography in Node is slow (see prev PR with benchmarks). Moving it to rust leads to about a 2x overall speedup
Change Summary

Add Rust toolchain to build
Add ed25519 signature verification in rust (via neon)
Move validations to use Rust to do signature verification

Merge Checklist
Choose all relevant options below by adding an x now or at any time before submitting for review

[X] PR title adheres to the conventional commits standard
[X] PR has a changeset
[X] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore)
[X] PR includes documentation if necessary.
[X] All commits have been signed

Additional Context
Benchmarks:
| Mainnet Hub | Rust hub (ed25519 in rust) | Impact
-- | -- | -- | --
validateMessage() | 780ms | 84ms | 9x
RPC Methods |   |   |
getFIDs | 0.03 ms | 0.03 ms | 1x
getCastsByFID | 0.07 ms | 0.08 ms | 1x
submitMessages | 98 ms | 32 ms | 3x
  |   |   |
Full sync from scratch | 129 min | 75 min | 1.7x
validateOrRevokeMessages |   |   |
(full DB iteration) | 66 min | 18 min | 3.6x

PR-Codex overview
Focus of this PR:
Add Rust to the toolchain and use Rust for ed25519 signature verification.
Detailed summary:

Added target/ and index.node to .gitignore in packages/core/src/addon/
Added 1.70 to packages/core/src/addon/rust-toolchain
Added export function nativeEd25519Verify(signature: Uint8Array, hash: Uint8Array, signer: Uint8Array): boolean; to packages/core/src/addon/addon.d.ts
Patched @farcaster/core in changeset/nervous-rivers-love.md
Updated tagline in apps/hubble/www/docs/index.md
Added src/addon/* to ignore in packages/core/rome.json
Updated test script in apps/hubble/package.json
Added Cargo.toml and src/addon/* in packages/core/src/addon/
Updated links in CONTRIBUTING.md
Updated target and added banner in packages/core/tsup.config.ts
Added import lib from "./index.node"; and nativeEd25519Verify function in packages/core/src/addon/addon.js
Updated content in apps/hubble/www/docs/intro/hubble.md
Updated Dockerfile.hubble
Added tests in packages/core/src/validations.test.ts
Updated validateMessage function in packages/core/src/validations.ts
Updated content in apps/hubble/www/docs/intro/install.md
Updated docker pull command in apps/hubble/www/docs/intro/install.md
Updated files and added dependencies in packages/core/package.json
Added ed25519_verify function in packages/core/src/addon/src/lib.rs
Updated Cargo.lock

The following files were skipped due to too many changes: packages/core/src/addon/Cargo.lock, yarn.lock

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

PR-Codex overview
Focus of this PR:
Add Rust to the toolchain and use Rust for ed25519 signature verification.
Detailed summary:

Added target/ and index.node to .gitignore
Added export function nativeEd25519Verify(signature: Uint8Array, hash: Uint8Array, signer: Uint8Array): boolean; to addon.d.ts
Patched @farcaster/core in rome.json
Updated tsup.config.ts with new imports and format
Added import lib from "./index.node"; and nativeEd25519Verify function to addon.js
Updated Dockerfile.hubble with installation of Rust and environment variables
Added ffi-napi and ref-napi as dependencies in package.json
Added Rust code in lib.rs for ed25519 signature verification
Updated validations.ts to use nativeEd25519Verify function
Updated Cargo.lock with new dependencies

The following files were skipped due to too many changes: packages/core/src/addon/Cargo.lock, yarn.lock

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

Nice, this looks good. Why did it improve the full iteration over messages so much? Or was this actually the validateOrRevokeMessages job?

Yes, it's not the iteration itself that's faster, but the validateOrRevoke is 10x faster

We need to add docs in a few places:
1. In `apps/hubble/www/docs/index.md` need change Typescript to Typescript + Rust

2. In `apps/hubble/www/docs/intro/hubble.md` need to do same as above

3. In `apps/hubble/www/docs/intro/install.md` we only show you how to install with Docker, but we should add a section at the bottom called Install from Source, which walks through all the steps needed.

@varunsrin - Added docs here: https://github.com/farcasterxyz/hub-monorepo/pull/1191/files#diff-25665ec3091d368ad2c870599c191d03a53aecd9de087de3eec696d0d6a40316.
Let me know if they look good


Security review PR - Collection of all "team project" feature related changes

Main Jira Ticket: VADC-618
New Features

Addition of "team project" access control to relevant cohort-middleware endpoints

thanks for the approval @k-burt-uch . I'll move just the new commits to a separate PR to master. Closing this one and keeping it for the record.


Fixed mewna/catnip#209

Added reason parameter to rest methods where it's possible to add a reason which is beeing displayed in audit log

I assume that you're better than @SamOphis and actually tested this?

I tested it at least at some endpoints

HEY. I do test to some extent. 😥


NativeModules.DevSettings is an empty object

Hi @gusgard. Thank you for this approach.
However, I am having an issue running this package in a React Native version 0.71.7 app.  I get the following error...

TypeError: Cannot read property 'setIsDebuggingRemotely' of null
TypeError: Cannot read property 'setIsDebuggingRemotely' of null

Triggered by NativeModules.DevSettings.setIsDebuggingRemotely(isDebuggingRemotely);.
https://github.com/gusgard/react-native-devsettings/blob/822cb8fdb28198fc73ae18ace33e540f81b50ee5/index.ts#L18
On logging NativeModules, it turns out to be an empty object.
Patched it to NativeModules.DevSettings?.setIsDebuggingRemotely(isDebuggingRemotely); to prevent the breaking error but the debugger is still not working. Any idea how I can go about this?
Came from react-native-debugger/issues/573

Hey, I'm not really sure because we are in 0.71.7 and it is working correctly.  Is this happening on both platforms iOS and Android? Could you share your Podfile.lock and package.json?
I wonder if you could try with something like setTimeout(() => require('react-native-devsettings'), 3 * 1000); (might be helpful if you are loading a lot of packages)

This approach did not work. I even tried loading the source code directly into my code and added the timeout there. Still no luck.
Here the pod file
pod_folder = Pathname.new(__FILE__).dirname.realpath

require File.join(File.dirname(`cd #{pod_folder} && node --print "require.resolve('expo/package.json')"`), "scripts/autolinking")
require File.join(File.dirname(`cd #{pod_folder} && node --print "require.resolve('react-native/package.json')"`), "scripts/react_native_pods")
require File.join(File.dirname(`cd #{pod_folder} && node --print "require.resolve('@react-native-community/cli-platform-ios/package.json')"`), "native_modules")

# Minimum 13.0 for Expo 47
platform :ios, "13.0"
prepare_react_native_project!

production = ENV["PRODUCTION"] == "1"

# If you are using a `react-native-flipper` your iOS build will fail when `NO_FLIPPER=1` is set.
# because `react-native-flipper` depends on (FlipperKit,...) that will be excluded
#
# To fix this you can also exclude `react-native-flipper` using a `react-native.config.js`
# ```js
# module.exports = {
#   dependencies: {
#     ...(process.env.NO_FLIPPER ? { 'react-native-flipper': { platforms: { ios: null } } } : {}),
# ```
flipper_config = ENV['NO_FLIPPER'] == "1" ? FlipperConfiguration.disabled : FlipperConfiguration.enabled
linkage = ENV['USE_FRAMEWORKS']
if linkage != nil
  Pod::UI.puts "Configuring Pod with #{linkage}ally linked Frameworks".green
  use_frameworks! :linkage => linkage.to_sym
end

target 'MyApp' do
  use_expo_modules!
  post_integrate do |installer|
    begin
      expo_patch_react_imports!(installer)
    rescue => e
      Pod::UI.warn e
    end
  end
  config = use_native_modules!

  # Flags change depending on the env values.
  flags = get_default_flags()

  use_react_native!(
    :path => config[:reactNativePath],
    :production => production,
    # Hermes is now enabled by default. Disable by setting this flag to false.
    # Upcoming versions of React Native may rely on get_default_flags(), but
    # we make it explicit here to aid in the React Native upgrade process.
    :hermes_enabled => flags[:hermes_enabled],
    :fabric_enabled => flags[:fabric_enabled],
    # Enables Flipper.
    #
    # Note that if you have use_frameworks! enabled, Flipper will not work and
    # you should disable the next line.
    :flipper_configuration => flipper_config,
    # An absolute path to your application root.
    :app_path => "#{Pod::Config.instance.installation_root}/.."
  )

  target 'MyAppTests' do
    inherit! :complete
    # Pods for testing
  end

  post_install do |installer|
    react_native_post_install(installer,
      # Set `mac_catalyst_enabled` to `true` in order to apply patches
      # necessary for Mac Catalyst builds
      :mac_catalyst_enabled => false
    )

    # Fix ios version errors in pods, like ReactCodeGen using 11.0 to use 13.0
    installer.pods_project.targets.each do |target|
      target.build_configurations.each do |config|
        config.build_settings['SWIFT_VERSION'] = '5.0'
        config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '13.0'
      end
    end

    __apply_Xcode_12_5_M1_post_install_workaround(installer)
    File.delete("./.xcode.env.local") if File.exist?("./.xcode.env.local")
  end
end

This approach did not work. I even tried loading the source code directly into my code and added the timeout there. Still no luck.
Here the pod file
pod_folder = Pathname.new(__FILE__).dirname.realpath

require File.join(File.dirname(`cd #{pod_folder} && node --print "require.resolve('expo/package.json')"`), "scripts/autolinking")
require File.join(File.dirname(`cd #{pod_folder} && node --print "require.resolve('react-native/package.json')"`), "scripts/react_native_pods")
require File.join(File.dirname(`cd #{pod_folder} && node --print "require.resolve('@react-native-community/cli-platform-ios/package.json')"`), "native_modules")

# Minimum 13.0 for Expo 47
platform :ios, "13.0"
prepare_react_native_project!

production = ENV["PRODUCTION"] == "1"

# If you are using a `react-native-flipper` your iOS build will fail when `NO_FLIPPER=1` is set.
# because `react-native-flipper` depends on (FlipperKit,...) that will be excluded
#
# To fix this you can also exclude `react-native-flipper` using a `react-native.config.js`
# ```js
# module.exports = {
#   dependencies: {
#     ...(process.env.NO_FLIPPER ? { 'react-native-flipper': { platforms: { ios: null } } } : {}),
# ```
flipper_config = ENV['NO_FLIPPER'] == "1" ? FlipperConfiguration.disabled : FlipperConfiguration.enabled
linkage = ENV['USE_FRAMEWORKS']
if linkage != nil
  Pod::UI.puts "Configuring Pod with #{linkage}ally linked Frameworks".green
  use_frameworks! :linkage => linkage.to_sym
end

target 'MyApp' do
  use_expo_modules!
  post_integrate do |installer|
    begin
      expo_patch_react_imports!(installer)
    rescue => e
      Pod::UI.warn e
    end
  end
  config = use_native_modules!

  # Flags change depending on the env values.
  flags = get_default_flags()

  use_react_native!(
    :path => config[:reactNativePath],
    :production => production,
    # Hermes is now enabled by default. Disable by setting this flag to false.
    # Upcoming versions of React Native may rely on get_default_flags(), but
    # we make it explicit here to aid in the React Native upgrade process.
    :hermes_enabled => flags[:hermes_enabled],
    :fabric_enabled => flags[:fabric_enabled],
    # Enables Flipper.
    #
    # Note that if you have use_frameworks! enabled, Flipper will not work and
    # you should disable the next line.
    :flipper_configuration => flipper_config,
    # An absolute path to your application root.
    :app_path => "#{Pod::Config.instance.installation_root}/.."
  )

  target 'MyAppTests' do
    inherit! :complete
    # Pods for testing
  end

  post_install do |installer|
    react_native_post_install(installer,
      # Set `mac_catalyst_enabled` to `true` in order to apply patches
      # necessary for Mac Catalyst builds
      :mac_catalyst_enabled => false
    )

    # Fix ios version errors in pods, like ReactCodeGen using 11.0 to use 13.0
    installer.pods_project.targets.each do |target|
      target.build_configurations.each do |config|
        config.build_settings['SWIFT_VERSION'] = '5.0'
        config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '13.0'
      end
    end

    __apply_Xcode_12_5_M1_post_install_workaround(installer)
    File.delete("./.xcode.env.local") if File.exist?("./.xcode.env.local")
  end
end

Could you share your Podfile.lock file? Are you having the same issue on Android ?

Try to install react-native-async-storage if you don't have it installed. It isnt in README but it is necessary to install it for this library to work


Sort ansible_play_hosts to ensure configure_ag script runs smoothly

Sort primary replica as the last in the list
Sort witness replica as the next-to-last in the list
Configure primary last because it has information about other replicas
Configure witness next-to-last to ensure that previous witness replica is
removed beforehand. Otherwise, script fails with 'only supports one replica

I hurried and merged without pushing my last commit. This work is to be continued in #99.


Example not correct with respect to multipart fields

In koa-body@4, the behavior of multipart has changed. The only example that exists for setting this up is here:
https://github.com/dlau/koa-body/blob/master/examples/multipart.js.
The only example showing how to access fields is a console.log (line 40) which is not correct (it indicates that fields is in ctx.request.fields, and in prior versions they were in ctx.request.body.fields). I ended up cracking open the source code of koa-body to figure out where fields are actually at (e.g. ctx.request.body is the fields). Simply updating the example would be a huge help so that it's not misleading.

This should have been addressed in #138.


Weekly Newsletter Generation

weekly newsletter generation code and corn job that run weekly on monday 00:00
for new threads list: titles which were created for the first time in the past week
for active post list: titles which have replies to older posts in the past week and those which are not already added to new thread's list
summary_of_threads_started_this_week: summary of the titles of new threads
added new fields: combined_summ_file_path along with file_path

Also, instead of overwriting the json every week, we should be archiving these newsletters.


Hardware requirements

What are minimum and recommended hardware requirements to run the model and to do training?
At https://huggingface.co/xai-org/grok-1 it is written:

Due to the large size of the model (314B parameters), a multi-GPU machine is required to test the model with the example code.

That multi-GPU machine mean exactly?
Also looks like that model weights itself are about 296.38 GB, so it would require more than 300 GB of storage. Should it be SSD or HDD will be enough?

many many gpus.

If you don't know what 300 GB of VRAM is, you have a lot to learn before trying to run this model.

could Jetson AGX Orin Developer Kit capable of running this monster model?

Question is whether the hardware requirements is an issue that can be fixed? Otherwise in my eyes it would seem that making it "Open source" only means making it available to businesses or in rare cases, individuals with the hardware to run it. Or was it just a publicity thing in relation to the OpenAi lawsuit...

looks like magnet download file is soooooo big, 256GB and 2.2% downloaded in progress.

@dabeckham  I don't know why people are staring it, no one has tested. Just goes viral. This is not release for us but only for Google, Microsoft and AWS etc.
Who can provide 300 + GPU memory???

@MuhammadShifa It will be possible to run this on the CPU once support is added to llama.cpp and someone releases 4-bit (or lower) quantized weights. You will need around 256 GB RAM, which is a lot more reasonable for a normal user than needing this much VRAM.

Looks like 8 GPUs with 80 GB VRAM each are not enough by themselves either: https://github.com/xai-org/grok-1/issues/125

looks like magnet download file is soooooo big, 256GB and 2.2% downloaded in progress.

Did you mean 296 GB?

ran it on 8x a100 80g with the code in this repo(no modification, i just added a loop to get input from terminal), using 524GB of vram during single batch inference with nearly no context(10~100 tokens input), speed is only 7 tokens per second
#168

Can run it on cloud hardware?

This looks interesting: #42 Speculations could be 96GB of vram if the model can be arranged to work at/with 4-bit quantization for the ggml library. Not sure how nicely ggml plays with jax though.

Just found this in relations to my last post on this thread: https://huggingface.co/eastwind/grok-1-hf-4bit
Looks to be about 90.2 GB on file if you add up the safetensor file shards from the hugging face eastwind repo.
There may be more overhead that requires a bit more memory to use for inference. But promising all the same.
I hope that grok-1 quants to 4 bit very well. Fingers crossed.

Following this calculation from https://www.substratus.ai/blog/calculating-gpu-memory-for-llm/ you would need

94,2 GB for the 4Bit Model
188,4 GB for the 8Bit Model

I've just stumbled upon this article from VMWare where you can open-source models in the cloud(s): https://www.vmware.com/products/vsphere/ai-ml.html#democratize

 Following this calculation from https://www.substratus.ai/blog/calculating-gpu-memory-for-llm/ you would need

94,2 GB for the 4Bit Model
188,4 GB for the 8Bit Model

I've just stumbled upon this article from VMWare where you can open-source models in the cloud(s): https://www.vmware.com/products/vsphere/ai-ml.html#democratize

Rough calculations is what I see this formula is for. Which is great for theory and rough plans.
But IRL the actual use of the model will likely have many tiny nuances from many different situations that can add up to change the picture enough to a point where it matters. For either Quantized or straight up open model use at any common float precision datatype.
It's a huge model. And I think that most who are paying attention are curious as a spectator. Which I admit is me as well. This is exciting and interesting stuff!
From what I'm seeing so far from others since my last post, what seems the most reachable, and performant for low mem use, could be roughly about 110 to 120+ GB for quantization. That's just for disk use and when loading the quantized model into mem. (See: https://huggingface.co/Arki05/Grok-1-GGUF for example.) Likely ballooning to a bit more in mem for basic forward passes.
Might be a tight fit for a Apple CPU inference with 128 GB of ram. But still asking a lot.

@MuhammadShifa It will be possible to run this on the CPU once support is added to llama.cpp and someone releases 4-bit (or lower) quantized weights. You will need around 256 GB RAM, which is a lot more reasonable for a normal user than needing this much VRAM.

The maximum amount of RAM i can squeeze into my AM5 board is 192GB of Ram at this moment.
Do you think it is feasible to get it running with this?

So I am Wondering if you can run this through lay man terms. Give me the run down of the computer parts for min. Running standards versus amazing running standards and what parts are interchangeable if i need to build this on the fly? is 8 GPUs really $4,000 for each gpu?
Is it possible to run a on a mac that is spec out to the max if not, sounds like a PC is the only answer. I need to figure out how to build the basic model of this as easiest as I can.


Proper handling local in hover

This PR implement back the Local hover information generation, which is fall back to a general case catch previously :
https://github.com/rust-analyzer/rust-analyzer/blob/9a44f627be0b3c49184e3ad594849f9b5ed78daa/crates/ra_ide/src/hover.rs#L173-L182

cc @kjeremy


Build tags on docker hub

Thanks for this cool project. Would it be possible to create docker builds on docker hub with tags corresponding to the source tags?

Hi @janschultecom.
Thank you for you comment!
Just to clarify: You want to have a Docker build for each individual tag? Today, we only use tags to identify different versions; so that would mean that we have Docker builds for each version. Have I understood you correct?
I could do some investigation on this during the weekend and come back to you after that. How does that sound to you?
Thanks!
Best regards,
Karl Dahlgren

Yes exactly. One docker build for each individual tag.
Right now one can only depend on the docker 'latest' image or build the image self from source. So relying on the latest is not a good option, because it might break things as soon as you do some breaking changes to the code and rebuild the latest image.

Closing this issue since it has been resolved. All the builds can now been found on the following page:  https://hub.docker.com/r/castlemock/castlemock/builds/


Merge ZMQ publisher to main branch.

This module allows a python script to publish current values to the network. This can be helpful when you have multiple computers running and you want to communicate latest values between them. For example:
Example usage
from NeuroLocoMiddleware.PubSubNetworking_ZMQ import Publisher

for t in loop:
    publisher.publish(topic = 'OSLV2.loopTimestamp',message="%6.2f" % t)
    publisher.publish(topic = 'OSLV2.filename', message = viconLogName)

Idk if it runs but it's well documented and has a test case :+1:


I want nunu.js

Issue title
Description

I want nunu.js not nunu.mn.js
please

Version

Platform

Web Version [X]
Windows [X]
Linux [X]

rename it  ?

Hi
I can add a nunu.js (not minified) build to the repo. Give me a sec.
Thanks a lot.

I can add a nunu.js (not minified) build to the repo. Give me a sec.

thank you

Hi
I can add a nunu.js (not minified) build to the repo. Give me a sec.
Thanks a lot.

And I found a bug with camera in orbit。when we add the camera into orbit ，and  adjust the camera angle. then i found the camera view is not the runtime view.

Hi,
In the camera Inspector, there is a property called "use camera" It's a checkbox. It has to be enabled in order for nunu to use that camera view on runtime.
Hope it helps,

thanks