Keeper backend doesn't work correctly with Custom Record Types

Describe the bug
When trying to access data from a custom record the fields are not processed correctly. The fields are identified by their type rather than their label. This leads to two problems

If you have duplicate field types, like say multiple text fields you get the error Following keys are duplicated <type>
If you don't have duplicates your fields are all called generic types like text or secret

To Reproduce
Steps to reproduce the behavior:

Create a couple custom record types one with duplicate field types one without
Try to access them with extract and no templating so you see all the keys in the resulting secret

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: test-secret
 namespace: test-secret
spec:
 refreshInterval: 1m
 secretStoreRef:
   name: ksm-secretstore
   kind: ClusterSecretStore
 target:
   name: test-secret
 dataFrom:
   - extract:
       key: uid

The one with the duplicates will give the error above, if you don't have duplicate field types then you'll get a secret with field names like text or secret instead of useful identifiers.
Expected behavior
Custom records with duplicate field types don't cause a complete failure. Records regardless of the field types are accessible via the label, since labels are free form it will probably require some form of manipulation like replace spaces with _

Hi @evilhamsterman ! Keeper is a community maintained provider - and as such @ppodevlabs is your best shot at getting this implemented 😄

Describe the bug When trying to access data from a custom record the fields are not processed correctly. The fields are identified by their type rather than their label. This leads to two problems

If you have duplicate field types, like say multiple text fields you get the error Following keys are duplicated <type>
If you don't have duplicates your fields are all called generic types like text or secret

To Reproduce Steps to reproduce the behavior:

Create a couple custom record types one with duplicate field types one without
Try to access them with extract and no templating so you see all the keys in the resulting secret

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: test-secret
 namespace: test-secret
spec:
 refreshInterval: 1m
 secretStoreRef:
   name: ksm-secretstore
   kind: ClusterSecretStore
 target:
   name: test-secret
 dataFrom:
   - extract:
       key: uid

The one with the duplicates will give the error above, if you don't have duplicate field types then you'll get a secret with field names like text or secret instead of useful identifiers.
Expected behavior Custom records with duplicate field types don't cause a complete failure. Records regardless of the field types are accessible via the label, since labels are free form it will probably require some form of manipulation like replace spaces with _

Hi, duplicated types should be allowed, what can not be duplicated is the label on the field. Could you please confirm you do not have duplicated labels?? As those are used as keys as stated in
remoteRef.property is equated to one of the following options:
- Fields: [Record's field's Type](https://docs.keeper.io/secrets-manager/secrets-manager/about/field-record-types)
- CustomFields: Record's field's Label
- Files: Record's file's Name
If empty, defaults to the complete Record in JSON format

Yes I do not have duplicate labels
I created a Custom Record type called "Duplicate type exampled" and a record of that type. Here's the sanitized json of the record object as returned by the Keeper API
{
  "record_uid": "recordid",
  "title": "Test record",
  "type": "Duplicate type example",
  "fields": [
    {
      "type": "text",
      "label": "Field 1",
      "value": [
        "foo"
      ]
    },
    {
      "type": "text",
      "label": "Field 2",
      "value": [
        "bar"
      ]
    }
  ],
  "custom": [],
  "version": 3,
  "shared": true,
  "client_modified_time": "2024-11-22T14:57:27",
  "user_permissions": [
    {
      "username": "myuser",
      "owner": true,
      "share_admin": true,
      "shareable": true,
      "editable": true
    }
  ],
  "shared_folder_permissions": [
    {
      "shared_folder_uid": "uuid",
      "reshareable": true,
      "editable": true,
      "revision": 12
    }
  ],
  "share_admins": [
    ""
  ],
  "revision": 12
}

When I create an ExternalSecret for that record  like this
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: test-secret
spec:
 refreshInterval: 1m
 secretStoreRef:
   name: ksm-secretstore
   kind: ClusterSecretStore
 target:
   name: test-secret
 dataFrom:
   - extract:
       key: recordid

it fails to sync the secret with the event invalid Secret. Following keys are duplicated text When it says text it means the duplicate text type field, the labels are different but the type is the same. This is shown by if I create another custom type with the fields
"fields": [
    {
      "type": "date",
      "label": "Date Active",
      "value": [
<PHONE_NUMBER>948
      ]
    },
    {
      "type": "date",
      "label": "Date of Birth",
      "value": [
<PHONE_NUMBER>326
      ]
    }
  ]

I get a similar error invalid Secret. Following keys are duplicated date
You properly support Custom FIELDS in a record but this is regarding Custom RECORD Types. As far as I can tell none of the builtin in record types have fields with duplicate types, but with custom records it is entirely possible to have more than one of the same type of field with the labels differentiating them.

Further more if it doesn't have duplicate types the name of the fields is the field type which may not be very useful in a custom record.
If I create a custom record with a text field labelled login and a multiline field labelled password to get around the duplicate field issue. I recognize this is a bit of a contrived but it's for demonstration. So now I have a record json
{
  "record_uid": "recordid",
  "title": "Wrong names",
  "type": "Wrong field names",
  "fields": [
    {
      "type": "text",
      "label": "Login",
      "value": [
        "hunter2"
      ]
    },
    {
      "type": "multiline",
      "label": "Password",
      "value": [
        "thatsthepasswordonmyluggage"
      ]
    }
  ],
  "custom": [],
  "version": 3,
  "shared": true,
  "client_modified_time": "2024-11-22T15:19:31",
  "user_permissions": [
    {
      "username": "user",
      "owner": true,
      "share_admin": true,
      "shareable": true,
      "editable": true
    }
  ],
  "shared_folder_permissions": [
    {
      "shared_folder_uid": "uuid",
      "reshareable": true,
      "editable": true,
      "revision": 13
    }
  ],
  "share_admins": [
    ""
  ],
  "revision": 13
}

It will successfully sync but the field names are not very useful they are just the names of the field types which my users may not be aware of because they don't have access to the record directly.
apiVersion: v1
data:
  multiline: dGhhdHN0aGVwYXNzd29yZG9ubXlsdWdnYWdl
  text: aHVudGVyMg==
immutable: false
kind: Secret
metadata:
  annotations: {}
  creationTimestamp: "2024-11-22T23:25:29Z"
  labels:
    reconcile.external-secrets.io/created-by: ee47526cfb7e5bedd021672be432e680
  name: test-secret
  namespace: it-github-runners
  ownerReferences:
  - apiVersion: external-secrets.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: test-secret
    uid: 76fcec59-7d7b-4643-968d-83663135fdfd
  resourceVersion: "4"
  uid: eeafdad1-da15-4d75-b74a-8b657816c621
type: Opaque

It would be better if rather than the field types for Custom Records it uses a sanitized version of the field label. Say lowercased and replace spaces with _ or something

Further more if it doesn't have duplicate types the name of the fields is the field type which may not be very useful in a custom record.
If I create a custom record with a text field labelled login and a multiline field labelled password to get around the duplicate field issue. I recognize this is a bit of a contrived but it's for demonstration. So now I have a record json
{
  "record_uid": "recordid",
  "title": "Wrong names",
  "type": "Wrong field names",
  "fields": [
    {
      "type": "text",
      "label": "Login",
      "value": [
        "hunter2"
      ]
    },
    {
      "type": "multiline",
      "label": "Password",
      "value": [
        "thatsthepasswordonmyluggage"
      ]
    }
  ],
  "custom": [],
  "version": 3,
  "shared": true,
  "client_modified_time": "2024-11-22T15:19:31",
  "user_permissions": [
    {
      "username": "user",
      "owner": true,
      "share_admin": true,
      "shareable": true,
      "editable": true
    }
  ],
  "shared_folder_permissions": [
    {
      "shared_folder_uid": "uuid",
      "reshareable": true,
      "editable": true,
      "revision": 13
    }
  ],
  "share_admins": [
    ""
  ],
  "revision": 13
}

It will successfully sync but the field names are not very useful they are just the names of the field types which my users may not be aware of because they don't have access to the record directly.
apiVersion: v1
data:
  multiline: dGhhdHN0aGVwYXNzd29yZG9ubXlsdWdnYWdl
  text: aHVudGVyMg==
immutable: false
kind: Secret
metadata:
  annotations: {}
  creationTimestamp: "2024-11-22T23:25:29Z"
  labels:
    reconcile.external-secrets.io/created-by: ee47526cfb7e5bedd021672be432e680
  name: test-secret
  namespace: it-github-runners
  ownerReferences:
  - apiVersion: external-secrets.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: test-secret
    uid: 76fcec59-7d7b-4643-968d-83663135fdfd
  resourceVersion: "4"
  uid: eeafdad1-da15-4d75-b74a-8b657816c621
type: Opaque

It would be better if rather than the field types for Custom Records it uses a sanitized version of the field label. Say lowercased and replace spaces with _ or something

This was built to work with non custom records that is why fields used the types instead of labels. I need to do some testing in order to verify this wont break the existing implementation. I will try to work on this next week.


document is not defined

I am using same structure of it, throws an error 'document is not defined'

Probably related to #12 if it doesn't have SSR support.


HDDS-4315. Use Epoch to generate unique ObjectIDs

What changes were proposed in this pull request?
In a non-Ratis OM, the transaction index used to generate ObjectID is reset on OM restart. This can lead to duplicate ObjectIDs when the OM is restarted. ObjectIDs should be unique.
For HDDS-2939 and NFS are some of the features which depend on ObjectIds being unique.
This Jira aims to introduce an epoch number in OM which is incremented on OM restarts. The epoch is persisted on disk. This epoch will be used to set the first 16 bits of the objectID to ensure that objectIDs are unique even after OM restart.
The highest epoch number is reserved for transactions coming through ratis. This will take care of the scenario where OM ratis is enabled on an existing cluster.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4315
How was this patch tested?
Will add unit tests in next commit.

General comment on using the epoch id that increments with every OM restart. This can get tricky.
If OM goes in crash restart loop then we have just 2^16 increments available which is 65K attempts. if it takes 1 secs for OM to comeback online we have 65 K secs worth epoch number or 20 hours of crash looping. This is very pessimistic view, it may take several seconds for OM to restart but it does show how

16 bit space can be insufficient for this scheme.
epoch need not be dependent on restart based increment. if it increments based on both of the following conditions
A) OM restart +
B) some object gets created after epoch id is incremented
then epoch may last longer. But even then 16 bit looks insufficient. What  if OM creates one object and restarts.

Thank you @linyiqun and @prashantpogde for the reviews.
Agree that setting aside 16 bits for epoch doesn't work for both the epoch as well as the transaction ids. 16 bits would not be enough to cover restarts and 40 bits might not be enough for transaction ids.
The new proposal is to have only 2 bits set aside for epoch. For non-Ratis OM, the transactionIndex will be saved in DB with every sync operation. When OM is restarted, this transactionIndex will be read from DB so that new transactions do not have clashing indices.
The epoch would let us distinguish objects created before and after this upgrade. This would help if someone needs to fix the duplicate objectIDs in existing clusters.
Thank you @bharatviswa504 and @prashantpogde for the offline discussion.

@prashantpogde, @linyiqun, @bharatviswa504, I have updated the PR with the discussed approach. Please review when you get a chance. Thanks.

Thank you @linyiqun and @prashantpogde for the reviews. I have addressed the comments. Please take a look.

Thanks @linyiqun and @prashantpogde for the reviews.


mysql(ticdc): remove returned error and modify the log in CheckIsTiDB

What problem does this PR solve?

Issue Number: close #11505
What is changed and how it works?
Description:
This PR refactors the CheckIsTiDB function in response to internal discussions. Previously, the function returned an error which was not necessary, as the external callers should not care about the error of the CheckIsTiDB function due to https://github.com/pingcap/tiflow/issues/11213.
Changes:

Removed the error return type from CheckIsTiDB.
Replaced the error log with a warning to handle cases where querying the TiDB version fails.
Updated comments to explain the background and rationale for the change.

Check List
Tests

Unit test

Questions
Will it cause performance regression or break compatibility?
No
Do you need to update user documentation, design documentation or monitoring documentation?
No
Release note
None

/retest

/check-issue-triage-complete

/pull-cdc-integration-kafka-test

/test dm-integration-test

/test cdc-integration-kafka-test

/test cdc-integration-kafka-test


Dropdown: Incorrect behavior when loading options dynamic.

Bug Report
Dropdown: Incorrect behavior when loading options dynamic.
Steps
I added custom options to the end and name it 'Load more'. When I click on the load more option I dynamically load options from backend and merge with the previous options. But position reset and move to the top.
The behavior reproduces when the selected element in the dropdown exists and is disabled.
I found a place where the resetting position.
If I have selected an item and it disabled after click 'load more'
It goes to https://github.com/Semantic-Org/Semantic-UI-React/blob/master/src/modules/Dropdown/Dropdown.js#L90
https://github.com/Semantic-Org/Semantic-UI-React/blob/master/src/modules/Dropdown/utils/getSelectedIndex.js#L33 we exclude the disabled item and therefore selectedIndex will be set to 0.
And in the https://github.com/Semantic-Org/Semantic-UI-React/blob/master/src/modules/Dropdown/Dropdown.js#L175 prevState.selectedIndex !== this.state.selectedIndex will be true and resetting postion
I removed this check and it started working as expected https://github.com/Semantic-Org/Semantic-UI-React/blob/master/src/modules/Dropdown/utils/getSelectedIndex.js#L33
Can you explain what's the check for? and maybe we can fix it?
Expected Result
The position in the view after added dynamic options not reset.
Actual Result
The position in the view after added dynamic options reset.
Version
2.0.1
Testcase

Please create a simple CodeSandbox with a repro case...

@layershifter  https://codesandbox.io/s/semantic-ui-example-forked-i33hg?file=/example.js

@layershifter  Could you please checking link above and to confirm this is a bug or not?

@layershifter  Could you please checking link above and to confirm this is a bug or not?


Powerline Theme (Blue)

Created a clone of the Powerline Multi-line Theme with some cosmetic changes and abbreviations of the current path.

Hi Again !
NOTE: I've contributed a lot to bash-it powerline so I take keen notice when anything related to it comes up ...
First thing I notice is that it looks like you may have started this branch some time ago, as there are a lot of features that are (now) present in the official powerline themes that are missing here, ie:

User-definable defaults for all variables
Recently-added segments: kubernetes, dirstack, etc.

After a bit of work, I was able to determine the main differences between your new theme and the current theme:
< # main powerline theme
> # new powerline-blue theme

< SCM_THEME_PROMPT_CLEAN_COLOR=${POWERLINE_SCM_CLEAN_COLOR:=25}
< SCM_THEME_PROMPT_DIRTY_COLOR=${POWERLINE_SCM_DIRTY_COLOR:=88}
< SCM_THEME_PROMPT_STAGED_COLOR=${POWERLINE_SCM_STAGED_COLOR:=30}
< SCM_THEME_PROMPT_UNSTAGED_COLOR=${POWERLINE_SCM_UNSTAGED_COLOR:=92}
---
> SCM_THEME_PROMPT_CLEAN_COLOR=2
> SCM_THEME_PROMPT_DIRTY_COLOR=2
> SCM_THEME_PROMPT_STAGED_COLOR=2
> SCM_THEME_PROMPT_UNSTAGED_COLOR=2
34a35,36
> SCM_GIT_SHOW_DETAILS=false
> SCM_GIT_SHOW_REMOTE_INFO=false
57c59
< CWD_THEME_PROMPT_COLOR=${POWERLINE_CWD_COLOR:=240}
---
> CWD_THEME_PROMPT_COLOR=39
59c61
< LAST_STATUS_THEME_PROMPT_COLOR=${POWERLINE_LAST_STATUS_COLOR:=52}
---
> LAST_STATUS_THEME_PROMPT_COLOR=196
87c89,91
< POWERLINE_PROMPT=${POWERLINE_PROMPT:="user_info scm python_venv ruby node cwd"}
---
> FG_COLOR=0
>
> POWERLINE_PROMPT=${POWERLINE_LEFT_PROMPT:="cwd scm python_venv ruby"}

Honestly, this doesn't look like enough change to warrant cloning the entire themes.
I'm the one who added the ability for the user to override every setting in the powerline themes - I did it specifically to enable users to highly-customize the themes without having to copy them.
With this ability, your changes can be applied locally in your .bash_profile (or whichever startup file you configure bash-it in) - Something like this should do it:
export BASH_IT_THEME='powerline'

export POWERLINE_SCM_CLEAN_COLOR=2
export POWERLINE_SCM_DIRTY_COLOR=2
export POWERLINE_SCM_STAGED_COLOR=2
export POWERLINE_SCM_UNSTAGED_COLOR=2
export SCM_GIT_SHOW_DETAILS=false
export SCM_GIT_SHOW_REMOTE_INFO=false

export POWERLINE_CWD_COLOR=39

export POWERLINE_LAST_STATUS_COLOR=196

export POWERLINE_PROMPT="cwd scm python_venv ruby"

Would you mind giving that a try and seeing how it goes?
If you really feel that that a lot of users would enjoy this specific combination of settings, you can still submit a new theme - It should be able to be reduced to just setting these values then including the original theme files, without having to clone them.
Lemme know what you think and also let me know if I missed something i.e if your effective changes were more than what I posted above.
-DF
NOTE: I am not a maintainer, just a contributor who volunteers to review PR's from time to time - My opinions may differ greatly from those of the official maintainers, so you are welcome to wait to hear from them before making any changes to your PR.

And hi again!
Definitely a lot has changed since I branched from it ;-)
I rebased it on the latest version of powerline and only needed to overwrite a few functions to get it working for me.
Thanks so much for your time and input!

Heyl thats a lot less changes, yay!
I didn't take a deep dive into the changes but I feel that the base file is still a bit larger than it needs to be.  I don't have any concerns on it tho.
My last actual comments on this would be:
As this is a rather personalized and "opinionated" configuration (i.e user cannot configure the segments or the clock format, for example), I would consider renaming this theme to something like "ucomesdag" which I feel more-clearly marks it as a personal theme, and gives you a bit more ownership on it, giving others a hint they might want to tag you if they were wanting make any changes.
There are a lot of themes that are named after their creator.  @nwinkler even has one :)
Also, you might want to add your screen shot (possible cropped) to the readme, so ppl can see what their getting if they try your theme.  I'm not actually sure what our process is for this (i.e where the images are placed), but I think other themes do it?)
Anyway, no more nagging from me - I'm just glad to see another powerline user :)
Good luck !
-DF

Thank you David for the feedback, appreciate it! If you are ok with it, I will rename it. I tried to quickly fish out all the difference, and I agree would be nice to trim down the base file more. When I arrive at that point I will probably merge it back in the original power-line theme.

Good discussion - thanks to both of you! A couple of comments:

I fully agree with @davidpfarrell - if this is just a variation of the Powerline settings (colours, etc.), then let's try to reuse as much as possible, i.e. just the changed settings in a new .theme.bash file.
Since it looks like you made a couple of changes to some of the functions in the .base.bash file, this looks like a separate theme to me - we should probably rename it in this case. It's not a real Powerline theme, but rather a separate theme that inherits some functionality from Powerline.
Do you have a high-level summary of the changes in the .bash.bash file that explain what your theme does differently? It would be interesting to see whether these could be covered in further switches for the Powerline theme, or whether they are really some other functionality.
Ah yes, the famous nwinkler theme... That was my first attempt at customizing my Bash prompt when I started to use Bash-it. Haven't used that theme in years - I'm using powerline-multiline these days, really happy with that.
Screenshots: Please don't include screenshots in commits, since they just add bloat to the repo size, which makes cloning a lot slower. Theme screenshots can be added to the Wiki (https://github.com/Bash-it/bash-it/wiki/Themes). There's a script (only works on macOS, though) discussed here to automatically create screenshots: https://github.com/Bash-it/bash-it/pull/260 - haven't used that in a while, but I know that it used to work.

In general, the direction for themes is that Bash-it has quite enough themes, if possible, we should steer people towards adding configurable changes to existing themes instead of creating new ones. It's too bad that we don't have usage tracking for themes, we don't know which themes are still in use and which ones are no longer used - would love to clean up the themes, but we can't do that since I'm afraid that we will remove things that people are using...

Will see if I can merge my changes back in the future.


Error: undefined function json_last_error_msg()

Hi, I get the following error when trying your script.
Fatal error: Call to undefined function json_last_error_msg() in /usr/www/libraries/httpProxyClass.php on line 360
I have error 416 from CloudFlare site (failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable)
Do you have any solution for this?

Interesting again can you tell me where you got this error, which site did you request with httpProxy?


DemoRoom: Baseball and PaintBrush are not stereo rendering

Hello,
I pull the latest code on GIT and do not change any code. However some objects are not stereo rendering. I can see all objects if  I close my right eye, but only see sky-box, text and gun if I close my left eye.
The same issue also exists on my meshes if I add my custom meshes with the following code:
m_engine->Get<EntityStore>()->Create<MyEntity>();
My MR device is HP Windows Mixed Reality Headset.
Thanks for your help.

The code base hasn't been changed for a while.  We cannot reproduce this issue on our side.   Is this a unique problem for this project or happens in other projects as well?

I tested today. This problem also happens in an other project: ControllerRenderSample.

@IngoChou why are you removing title and content from your issues?


Update GC-MS instrument name

Guidelines
Soft Schema Freeze
The nmdc-schema and berkeley-schema-fy24 schemas are under a soft freeze, which means changes should not be made that have any downstream implications. To ensure this, all PRs created during the freeze will be closely reviewed with every component of the NMDC system in mind.
Reviewers
To ensure no changes are made unexpectedly, PR creators will use this PR template to tag and notify all task coordinators. Review should be specifically requested from all Berkeley Schema Roll Out task coordinators that you expect to be affected by this PR.
We expect task coordinators to review PRs and provide feedback/approval within 1 week of when they are identified as reviewers.
PRs will NOT be merged until all task coordinators (or their delegates) have approved it; either here on GitHub (via "Review changes > Approve" or an equivalent comment) or verbally.
Expedition, questions, and discussion can happen at any meeting.
Delays in review & merging should be addressed in meetings or with NMDC leadership.

If you expect the changes toimpact this component...
...request a reviewfrom this person

MetadataSchema
@mslarae13

RuntimeMongo databaseDatabase migrations
@eecavanna,who will pull in@shreddd as needed

PostgresIngest
@naglepuff

Data Portal
@aclum

Workflows: MG & MT
@mbthornton-lbl

Workflows: MetaB & NOM
@corilo

Workflows: LipidO
@kheal

Workflows: MetaP
@SamuelPurvine

ETL code
@sujaypatil96

Jupyter notebooks
@brynnz22

PR Information
What type of PR is this? (check all applicable)

[ ] Refactor
[ ] Feature
[ ] Bug Fix
[ ] Optimization
[ ] Documentation
[ ] Schema change: Structure and content

created, updated, or deleted a class, slot, or enum
changed whether a slot is multivalued
changed the way a slot is assigned to a class
changed the permissible_values of an enum
etc.

[ ] Schema change: Cleanup and preparation

updated the description of a class, slot, or enum
updated the mappings of a class, slot, or enum to an ontology
added an enum for future use (it is not in the range of any slot)
etc.

Description

PRs should be small and concise.
Aim to create small, focused pull requests that fulfill a single purpose. Smaller pull requests are easier and faster to review and merge, leave less room to introduce bugs, and provide a clearer history of changes.

Update the GC-MS instrument name to be more descriptive (to Agilent GC-MS 1 (2009)) Since the current data in Mongo saysAgilent_GC_MS_01 and the model we have in the file is 7980A, I matched it to that instrument name in Nexus @corilo
Related Issues

All PRs should relate to or fix an issue(s). Please identify the issue(s) below.

Related Issue(s): #
Fixes: #

Did you add/update any tests?

[ ] Yes
[ x] No (Add a justification below)
no need
[ ] I need help with writing tests

Could this schema change make it so any valid data becomes invalid?

This is a question about what the schema allows. It is not a question about what happens to exists in the NMDC database right now.
Example: If, in this PR branch, you renamed a slot from foo to foo_bar, the answer to this question would be "yes," even if nothing in the NMDC database currently uses the foo slot.
More examples: slot or class name changes, changes to a slot's multivalued state, changes to a slot's range (e.g. string to integer), changes to slot assignments to classes, changes to an enum's permissible_values

[ ] Yes (A migrator is required)
[x ] No
[ ] I need help determining this

If you answered "Yes", does this PR branch include that migrator?

[ ] Yes
[ ] No, this PR is incomplete and I need help writing the migrator

Does this PR have any downstream implications?
It impacts a migrator, but should not have an downstream implication other than impacted what the migrator matches the current instrument in Mongo to this new name.

Examples: any change here that requires a change to workflows, workflow automation, the Mongo-to-Postgres ingest process, Jupyter notebooks, the Runtime, etc.

[ ] Yes (Explain below)
[ s] No

@brynnz22 can you add a comment on why this was closed an not merged? The work won't be done.. or the PR will be re-done with the new schema?

We decided to do this via a changesheet rather than migration.  See https://github.com/microbiomedata/nmdc-schema/issues/2208


Adds basic implementations for vpc-classic-link and vpc-classic-link-dns-support - Redo

Update to https://github.com/spulec/moto/pull/2404, didn't want to step over his branch

Coverage decreased (-0.002%) to 93.668% when pulling 578e8b03951336dfaece46ad933d40768a17a475 on catherinetcai:feature/ec2-vpc-classic-link into b19abbc63ec2dbaa7802c783942e5925f84aa3b2 on spulec:master.


address #16743

Two distinct commits for the issues noted in #16743.

explicitly removes extraneous padding on a elements in vertical-tabs elements
removes cursor/hover behavior for vertical-tabs__tab class, which is predominately used on a elements which have the desired behavior directly.

removes cursor/hover behavior for vertical-tabs__tab class, which is predominately used on a elements which have the desired behavior directly.

I think this probably breaks the effect on mobile where the entire element is clickable, not just the link?

I think this probably breaks the effect on mobile where the entire element is clickable, not just the link?

Can you say where? I'm testing it with vertical tabs on my phone and not seeing a difference.

superseded by #16746


Is there a way to avoid creating Secret Manager secret

The newrelic-lambda cli allows the use of --disable-license-key-secret to avoid using Secret Manager for storing the New Relic License Key (https://docs.newrelic.com/docs/serverless-function-monitoring/aws-lambda-monitoring/enable-lambda-monitoring/enable-serverless-monitoring-using-lambda-layer).
Is there a similar flag available for this serverless plugin?

Linked PR: https://github.com/newrelic/serverless-newrelic-lambda-layers/pull/116


Support to use functions for default response headers

It would be great if I could set up a default reply header that is generated via a function, the first 2 use-cases I can think of are setting the Content-Length header and a Date header.
scope.defaultReplyHeaders({
  'Content-Length': function (req, res) {
    return res.body.length;
  },
  'Date': function () {
    return (new Date()).toUTCString();
  }
});

@dominicbarnes That would be nice, I would be open to accepting a tested PR with that :)


mDNS problem (CON-465)

What is going on in this attached log?
mDNS.log
There are two commissions going on. I am using Android.

First commission is being done by Google. They are doing it so that they can set the wifi ssid/pw into the device.
Second commission is my app. This second commission is what triggered the first one since the Google lib knew it had to insert the SSID/pw into the device.

I am making my own certificates so that could be part of the problem.
It got to this point and it died:
I (64213) chip[FP]: NOC chain validation successful
I (64213) chip[FP]: Added new fabric at index: 0x2I (64213) chip[FP]: Assigned compressed fabric ID: 0x2A3116CDB0CC76FE, node ID: 0x0000000000000002
I (64213) chip[TS]: Last Known Good Time: 2023-05-02T10:15:03
I (64223) chip[TS]: New proposed Last Known Good Time: 2021-06-10T00:00:00
I (64223) chip[TS]: Retaining current Last Known Good TimeI (64243) chip[ZCL]: OpCreds: ACL entry created for Fabric index 0x2 CASE Admin Subject 0xDEDEDEDE0000001D
I (64243) chip[DIS]: Advertise operational node 4284C2F5A80B5F7C-8E11AC440B94CF84
I (64243) chip[DIS]: CHIP minimal mDNS configured as 'Operational device'; instance name: 4284C2F5A80B5F7C-8E11AC440B94CF84.
I (64253) chip[DIS]: mDNS service published: _matter._tcp
I (64253) chip[DIS]: Advertise operational node 2A3116CDB0CC76FE-0000000000000002
I (64253) chip[DIS]: CHIP minimal mDNS configured as 'Operational device'; instance name: 2A3116CDB0CC76FE-0000000000000002.
I (64273) chip[DIS]: mDNS service published: _matter._tcp
I (64273) chip[ZCL]: OpCreds: successfully created fabric index 0x2 via AddNOC
I (64273) chip[EM]: <<< [E:29773r S:3097 M:104859328 (Ack:187132537)] (S) Msg TX to 2:FFFFFFFB00000000 [76FE] --- Type 0001:09 (IM:InvokeCommandResponse)
I (64273) chip[IN]: (S) Sending msg 104859328 on secure session with LSID: 3097
E (64283) chip[DL]: Long dispatch time: 325 ms, for event type 3
I (64283) chip[EM]: >>> [E:29772r S:3097 M:187132538 (Ack:104859326)] (S) Msg RX from 2:FFFFFFFB00000000 [76FE] --- Type 0000:10 (SecureChannel:StandaloneAck)
I (64333) chip[EM]: >>> [E:29773r S:3097 M:187132539 (Ack:104859328)] (S) Msg RX from 2:FFFFFFFB00000000 [76FE] --- Type 0000:10 (SecureChannel:StandaloneAck)

>>> Why is only one mDNS visible? 0002 is missing

It died because the second mDNS did not become visible
2023-05-02 14:36:04.496 10675-11601 EM                      com.lowpan.m2                        D  Flushed pending ack for MessageCounter:104859328 on exchange 29773i
2023-05-02 14:36:04.676 10675-11601 DIS                     com.lowpan.m2                        D  Checking node lookup status after 200 ms
2023-05-02 14:36:04.725 10675-11564 NsdService...ndResolver com.lowpan.m2                        D  Service discovery started. regType: _matter._tcp
2023-05-02 14:36:04.727 10675-11564 NsdService...ndResolver com.lowpan.m2                        D  onServiceFound: found service not a target for resolution, ignoring name: 4284C2F5A80B5F7C-8E11AC440B94CF84, type: _matter._tcp., host: null, port: 0, network: 108, txtRecord:
2023-05-02 14:36:04.727 10675-11564 NsdService...ndResolver com.lowpan.m2                        D  onServiceFound: found service not a target for resolution, ignoring name: 2A3116CDB0CC76FE-0000000000000002, type: _matter._tcp., host: null, port: 0, network: 108, txtRecord:
2023-05-02 14:36:04.821 10675-11564 NsdService...ndResolver com.lowpan.m2                        D  onServiceFound: found service not a target for resolution, ignoring name: C7944E937AA07DD5-0083161AE5399201, type: _matter._tcp., host: null, port: 0, network: 108, txtRecord:
2023-05-02 14:36:05.028 10675-11564 NsdService...ndResolver com.lowpan.m2                        D  onServiceFound: found service not a target for resolution, ignoring name: 950545243CFE2D15-00000000F1077CC7, type: _matter._tcp., host: null, port: 0, network: 108, txtRecord:
2023-05-02 14:36:05.029 10675-11564 NsdService...ndResolver com.lowpan.m2                        D  onServiceFound: found service not a target for resolution, ignoring name: C7944E937AA07DD5-00867AF748BEBA01, type: _matter._tcp., host: null, port: 0, network: 108, txtRecord:
2023-05-02 14:36:05.033 10675-11564 NsdService...ndResolver com.lowpan.m2                        D  onServiceFound: found service not a target for resolution, ignoring name: C7944E937AA07DD5-008462EE7296AD71, type: _matter._tcp., host: null, port: 0, network: 108, txtRecord:
2023-05-02 14:36:05.034 10675-11564 NsdService...ndResolver com.lowpan.m2                        D  onServiceFound: found service not a target for resolution, ignoring name: 9CC0373667C4F408-204B50A898A38F65, type: _matter._tcp., host: null, port: 0, network: 108, txtRecord:
2023-05-02 14:36:09.494 10675-11607 NsdService...ndResolver com.lowpan.m2                        D  Service discovery timed out after 5000 ms
2023-05-02 14:36:09.497 10675-11564 NsdService...ndResolver com.lowpan.m2                        I  Discovery stopped: _matter._tcp
2023-05-02 14:36:34.523 10675-10675 NsdManager...ceResolver com.lowpan.m2                        D  resolve: Timing out
2023-05-02 14:36:49.518 10675-11601 DIS                     com.lowpan.m2                        D  Checking node lookup status after 45042 ms
2023-05-02 14:36:49.518 10675-11601 DIS                     com.lowpan.m2                        E  OperationalSessionSetup[1:0000000000000002]: operational discovery failed: ../../src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:114: CHIP Error 0x00000032: Timeout
2023-05-02 14:36:49.518 10675-11601 CTL                     com.lowpan.m2                        D  Device connection failed. Error ../../src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:114: CHIP Error 0x00000032: Timeout
2023-05-02 14:36:49.520 10675-11601 BaseCompletionListener  com.lowpan.m2                        D  onCommissioningStatusUpdate nodeId [2]  stage [FindOperational]  errorCode [50]
2023-05-02 14:36:49.520 10675-11601 CTL                     com.lowpan.m2                        D  Error on commissioning step 'FindOperational': '../../src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:114: CHIP Error 0x00000032: Timeout'
2023-05-02 14:36:49.520 10675-11601 CTL                     com.lowpan.m2                        E  Failed to perform commissioning step 18
2023-05-02 14:36:49.520 10675-11601 CTL                     com.lowpan.m2                        D  Going from commissioning step 'FindOperational' with lastErr = '../../src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:114: CHIP Error 0x00000032: Timeout' -> 'Cleanup'
2023-05-02 14:36:49.520 10675-11601 CTL                     com.lowpan.m2                        D  Performing next commissioning step 'Cleanup' with completion status = '../../src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:114: CHIP Error 0x00000032: Timeout'
2023-05-02 14:36:49.522 10675-11601 BaseCompletionListener  com.lowpan.m2                        D  onCommissioningStatusUpdate nodeId [2]  stage [Cleanup]  errorCode [0]
2023-05-02 14:36:49.522 10675-11601 CTL                     com.lowpan.m2                        D  Successfully finished commissioning step 'Cleanup'
2023-05-02 14:36:49.523 10675-11601 BaseCompletionListener  com.lowpan.m2                        D  onCommissioningComplete(): nodeId [2] errorCode [50]
2023-05-02 14:36:49.527 10675-10675 AppCommiss...gRequested com.lowpan.m2                        E  onCommissioningRequested() failed
                                                                                java.lang.IllegalStateException: Commissioning failed with error code [50]
                                                                                at com.lowpan.m2.chip.ChipClient$awaitCommissionDevice$2$1.onCommissioningComplete(ChipClient.kt:293)
                                                                                at chip.devicecontroller.ChipDeviceController.onCommissioningComplete(ChipDeviceController.java:335)
2023-05-02 14:36:49.535 10675-10675 AppCommiss...gRequested com.lowpan.m2                        D  Commissioning: commissioningServiceDelegate.sendCommissioningError() succeeded
2023-05-02 15:03:52.277 10675-14306 TrafficStats            com.lowpan.m2                        D  tagSocket(130) with statsTag=0xffffffff, statsUid=-1

I added the CAT tag and it made no difference.

The problem is that Android and ESP32 are disagreeing on the compressed fabric ID.

I've figured out that I need to build a NOCChainIssuer class.

Closing this here


Resolve Klocwork issues found in 4.2.1a

See attachment

Imported from trac issue 218.  Created by jhageman on 2019-06-03T13:31:11, last modified: 2019-08-14T14:11:46


Error during Sydney 10.4.1 compiling

Hi Laex,
I've tried to build Delphi-OpenCV-Class with Syndey 10.4.1 following your README info.
When I try to build the package I obtain this error message:

Try in the file "Delphi-OpenCV-Class\source\cpp.utils.pas" on line 114 to change
TPtr<T:record> = record
on the
TPtr = record

Perfect now compile :)
Another error during design package compilation:
dibhdr := @buf;
to
dibhdr := pBITMAPINFOHEADER(@buf);

Yes, I corrected that too.


⚠ Bofrost.be has degraded performance

In 195169c, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2377 ms

Resolved: Bofrost.be performance has improved in 1643ab7.


evaluating on MiniImageNet does not achieve good performance by using pretrained model.

Hi,
Thanks for your interest in our work.
The pre-trained model needs to be fine-tuned by meta-learning to get the best performance.
The results using the pre-trained model directly for evaluation is also reported in the paper, please check it for details.

Thanks, I will try to fine-tune on the MiniImageNet.

@Lyrics1988 Hi, I fine-tuned the pre-trained model(mini_imagenet_ndf192_rkhs1536_rd8_ssl_cpt.pth) on MiniImageNet, but only got the follow results:
Val Best Acc 0.7359, Test Acc 0.6301
Test Acc 0.6301 + 0.0020

@QWERDF007 Hi, thanks for your feedback.
Please try to train the model again, I've re-train the network and I got:
Val Best Acc 0.7254, Test Acc 0.6385
Test Acc 0.6385 + 0.0020
This is within the error range of the reported result.

Thanks, I will try it again. By the way, is it possible to train a 10-shot 5/10-way with 2080Ti, which only has 11GB GPU memory?

@Lyrics1988  Thanks, I will try it again.
By the way, is it possible to train a 10-shot 5/10-way with 2080Ti, which only has 11GB GPU memory?


Request your trained model

Congratulations on your winning. And thanks for sharing your work. I'm very interested in the YouTube-8M Large-Scale Video Understanding project.  Can you send me your trained model. My<EMAIL_ADDRESS>Thanks.

Hi,
Yes sure, I just realized Google had released their code for feature extraction.
I will update the github repo with the pretrained best model I have (If I find it back ahha :) ).

Here it is: https://www.rocq.inria.fr/cluster-willow/amiech/pretrainedmodel.zip
This is the model for :
python inference.py --output_file=test-gatednetvladLF-256k-1024-80-0002-300iter-norelu-basic-gatedmoe.csv --input_data_pattern="$path_to_features/test*.tfrecord" --model=NetVLADModelLF --train_dir=gatednetvladLF-256k-1024-80-0002-300iter-norelu-basic-gatedmoe --frame_features=True --feature_names="rgb,audio" --feature_sizes="1024,128" --batch_size=1024 --base_learning_rate=0.0002 --netvlad_cluster_size=256 --netvlad_hidden_size=1024 --moe_l2=1e-6 --iterations=300 --learning_rate_decay=0.8 --netvlad_relu=False --gating=True --moe_prob_gating=True --run_once=True  --top_k=50

Thanks for your sharing. @antoine77340


Allow users to configure compiler-settings

Add input parameter solcSettings to function compile.
The default value for this parameter is {optimizer: {enabled: true, runs: 200}}, identical to the Solidity Compiler's default configuration.
This feature is in fact very important, because certain compiler-settings may yield the difference between successful compilation and unsuccessful compilation.
Thus, without this feature, users might find themselves being able to compile their contracts but unable to execute solidity-docgen, as a result of the latter failing to compile the exact same contracts.
Hence this feature can also be considered as a bug fix.
You can see a living-example of the scenario above in this contract (note that it is currently on a branch and will be moved to master in the future, so use this link if the previous one is broken).
The compilation error Stack too deep is emitted only when optimization is disabled.
TODO: Add a description of the solcSettings configuration parameter in the README file.

@barakman I've fixed the compilation errors. Can you confirm that the code on this branch works well for you?

@frangio:
Whenever I want to add or fix something in solidity-docgen, I first change the Javascript code downloaded to my local machine via npm install, in order to test it against my local repository.
Then, in order to commit it here via PR, I kinda guess what I need to do in your Typescript code (assuming that ts extension stands for Typescript).
I don't really know any way for testing those changes (in fact, I don't know anything about Typescript in general, except for the few changes that I've committed here recently).
So technically, in order to verify your changes, I'd need you to release a version.
Otherwise, I'd have to import your changes to my local machine, which is of course prone to errors.
I will report back in a few minutes (hopefully), but I guess it's not gonna be a perfect type of verification.
Thanks

@frangio:
I'm sorry, but I'm not really sure how to embed the following changes that you've made into the solidity-docgen code on my local machine:

In file src/docgen.ts (https://github.com/OpenZeppelin/solidity-docgen/pull/45/files#diff-b2ef23e44847b71971532bdf81be1eec)
In file src/cli.ts (https://github.com/OpenZeppelin/solidity-docgen/pull/45/files#diff-cd02b9e0b731a6a1dedefed142f4419b)
Would it be possible for you to release some sort of alpha version with all the recent changes, so that I can test it quickly without doing any "local tweaks"?
Thanks

You should be able to test the code on this branch by running
npm install github:barakman/solidity-docgen#patch-2

@frangio:
OK, so on my local repo, I have verified:

solidity-docgen completes successfully when I pass "--solc-settings=" + JSON.stringify({optimizer: {enabled: true, runs: 200}}).
solidity-docgen fails with CompilerError: Stack too deep when I pass "--solc-settings=" + JSON.stringify({optimizer: {enabled: false, runs: 200}}).
solidity-docgen completes successfully when I omit the --solc-settings flag (thus, testing that the default value for this flag works as expected).

So I am confirming that this PR works well for me.
Thank you very much for helping to resolve this issue.
When would it be possible for you to release an official version which includes this PR, as well as PR #46?
Thanks

Nice! Thank you for testing @barakman!

I've published 0.3.0 to npm including this PR.

@frangio:
Thank you very much for this :)
Unfortunately, patch-3 (extending the HBS functionality) was not properly tested before merging to master (my fault for approving your two commits in PR #46 without properly testing them).
Please see PR #48.
My apologies.
Thanks


IIS custom error pages do not work with ANCM

The IIS custom error page feature allows IIS to provide/replace the response body for specific status codes. By default it will do this for 401,403,404,405,406,412,500,501,502.  However, this does not work with ANCM/Kestrel, all configured error pages are bypassed and the original body from the application is returned to the client.
Questions:

Do we want to support IIS custom error pages with Asp.Net Core? We have a replacement feature in Diagnostics called StatusCodePages. We also have multiple 500 pages.
If yes, should it be opt-in or opt-out? If it were opt out you would be unable to see any of the Asp.Net diagnostics pages by default.

@DamianEdwards @shirhatti @pan-wang @davidfowl

Note we see the same behavior in Azure.

@blowdart

I vote for making this an opt-in feature. I enjoy keeping as much server-ish work within Kestrel as possible ... even secretly hoping that Kestrel could get the needed features to make it a full-fledged, standalone server for WS one day.
If you made this an opt-out feature, it feels like that would be leaning in the opposite direction. I hope every decision that comes down to choosing default behavior between IIS and Kestrel leans in the Kestrel direction.

@muratg This was by design. We don't see enough people asking for this to be done.
I'm leaving this issue open to gather feedback, but right we don't plan on changing behavior

This issue was moved to aspnet/AspNetCoreModule#31


proposal: cmd/go: build tag in filename suffix for matching of syso files

Motivation
We have several Go packages that targets multiple platforms, including macOS, iOS, tvOS, iOS-simulator, tvOS-simulator, windows, linux, etc.  The package has one or more syso file dependencies.
Syso files can be selected for a build by a GOOS/GOARCH filename suffix but not build tags, unlike Go source files that use // +build or //go:build lines.  If we ignore tvOS for the moment, some time ago we could build a syso file for darwin/amd64 and include that in both macOS and iOS-simulator builds just fine.  With recent Xcode 12, that approach no longer works:
ld: building for iOS-simulator, but linking in object file built for macOS, file '/var/folders/h7/r_43shxn71v0pq_gx72f57ph0000gn/T/go-link-172505736/000035.o' for architecture x86_64

The situation worsens when we support tvOS (+ the tvOS-simulator), and the new ARM-based Apple Silicon Macs because we can no longer use a GOOS/GOARCH filename build constraint to uniquely select the appropriate syso variant.
Workaround
In #38485 @cherrymui suggested using build-tags that selectively import sub-packages containing the associated syso file(s).  That can work, but is messy as outlined next.
In more detail, the idea is to write a Go source file with a // +build or //go:build line that is conditional on one or more tags of interest.  The conditional build of said source file then imports another sub-package that contains the syso file to use in the build.
For every syso file, it would require one additional sub-package and two additional Go source files.  (One for with the conditional build line, and another to define the otherwise empty package containing the syso file.)  This balloons when there might be a Debug and Release version of each syso file, requiring 4n Go source files and 2n sub-package directories for n platforms.
A better approach would be to support matching build tags in syso filenames.  The syso files can be placed directly into a package, for a better developer experience.
Proposal
This proposal is to extend filename build constraints to include build tags with a _#tag pattern in the filename suffix, before any GOOS and/or GOARCH patterns.
Multiple tags can be supported with _#tag1_#tag2_#tag3, where an AND matching condition (tag1 && tag2 && tag3) is used for selection.
Where no tag suffix items are present, the behavior is the same as it is now.
Example
We can use the user-defined build tags macosx (macOS), iphoneos (iOS), iphonesimulator (iOS-simulator), appletvos (tvOS), appletvsimulator (tvOS-simulator), androidos (Android), linuxos (Linux) on Macs, iOS/tvOS devices, Linux, Android, and Windows as follows:

lib_#macosx_amd64.syso for macOS on an Intel Mac
lib_#macosx_arm64.syso for macOS on an Apple Silicon Mac
lib_#iphonesimulator_amd64.syso for iOS-simulator on an Intel Mac
lib_#iphonesimulator_arm64.syso for iOS-simulator on an Apple Silicon Mac
lib_#appletvsimulator_amd64.syso for tvOS-simulator on an Intel Mac
lib_#appletvsimulator_arm64.syso for tvOS-simulator on an Apple Silicon Mac
lib_#iphoneos.syso for an iOS device (note: no GOARCH)
lib_#appletvos.syso for an Apple TV device (note: no GOARCH)
lib_#linuxos_amd64.syso for Linux desktop 64-bit Intel
lib_#linuxos_arm64.syso for Linux desktop 64-bit ARM
lib_#androidos_arm64.syso for Android 64-bit ARM
lib_windows_amd64.syso for Windows 64-bit Intel

Notes

While this proposal could work for Go source files and syso files alike, the current proposal is for syso files only since, unlike Go source files, there is no alternative other than the workaround above.

I have tested the use of the "#" character in filenames on macOS, Windows, and Linux filesystems, and didn't see a problem with using that character for this purpose.

cc @eliasnaur

This seems like a lot of work to support .syso files. Why do you need them? Is it to avoid the need for SDKs and system compilers/linkers? If so, it seems to me the effort is better spent on that. Shameless plug: https://github.com/golang/go/issues/38917.

Kind of an odd idea, but could fat .syso files solve the problem? Not sure whether those are even a thing, but if it were possible then that would allow us to transfer the problem to the loader.
The loader can distinguish amd64 vs arm64. Can it distinguish the other axes, like simulator vs. non-simulator?

This seems like a lot of work to support .syso files. Why do you need them? Is it to avoid the need for SDKs and system compilers/linkers? If so, it seems to me the effort is better spent on that. Shameless plug: #38917.

A lot of work for a contribution for a CL to address this, or a lot of work for a 3rd party developer who might make use of this?

On the former, I'm happy to make the contribution; I don't think it'd be anywhere near as complex as #33139 was.  At a high-level, the proposal is very simple: support selection of syso files at build time based on user-defined build tags.  The proposal does not suggest adding any pre-defined build tags into gc.
On the latter, it's not much work: we use a Makefile to generate each platform object using a cross compiler.  The resulting syso files are committed to the project using Git-LFS and are thereafter "transparent" to Go builds, subject to the limitations that motivated this issue.

We use syso files for two reasons:

To have a "go gettable" package that statically links a library that is called from cgo.  See this public example.  This is probably what you had in mind w.r.t. #38917.
To call precompiled functions directly from Go (without cgo and its associated overhead), that were written in GAS, or C with SIMD intrinsics and/or auto-vectorization.

Kind of an odd idea, but could fat .syso files solve the problem? Not sure whether those are even a thing, but if it were possible then that would allow us to transfer the problem to the loader.
The loader can distinguish amd64 vs arm64. Can it distinguish the other axes, like simulator vs. non-simulator?

That's actually a great suggestion.  I just tried it and unfortunately hit this:
fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: lib_device.o and lib_sim.o have the same architectures (arm64) and can't be in the same fat output file

Clang LD can handle "fat objects", but each fat object can only host dissimilar architectures, even if each component is built for a different platform.  This becomes pretty clear when you look at the Mach-O fat header format: each slice is keyed on a CPU and sub-CPU architecture magic.
That means we cannot, for example, have a fat object that targets both iOS (arm64 device) and the iOS-simulator (arm64 Apple Silicon Mac); or iOS (arm64 device) and tvOS (arm64 device) and/or the respective simulators.

To call precompiled functions directly from Go (without cgo and its associated overhead), that were written in GAS, or C with SIMD intrinsics and/or auto-vectorization.

Currently there isn't anything stop you from doing this. But this is not really "supported".

To call precompiled functions directly from Go (without cgo and its associated overhead), that were written in GAS, or C with SIMD intrinsics and/or auto-vectorization.

Currently there isn't anything stop you from doing this. But this is not really "supported".

@cherrymui, you wrote something similar back in July last year.  As I replied back then, the Go Wiki documents this pattern since at least 2014 [1].  I spent a lot of time to contribute a CL to "support" that late last year, and I'm offering to support it again here.
Separate to the motivation for the proposal, I'd be grateful for any comments or support for the concrete use of #tags in a syso filename suffix.  With some blessing, or at least no objections, I'd be happy to start working on a CL.
[1] See wiki commit 5bc444, which was "import from Google Code": it was probably written much earlier than that.

I know the Wiki page mentions it, as a "trick". But I wouldn't say it is "supported".
I agree that we probably don't want to intentionally break it, and probably will keep it working if it doesn't take much effort. But I don't think this kind of use is a good motivation for new proposals.

I think it might be a good idea to have some mechanism to match build tags for syso files (although I don't necessarily agree with the motivation).
I think the proposal needs to be a little clearer about the matching semantics. Is the tag after the # character must be present, so a build with no tags specified doesn't match any?
If we do this, I think supporting multiple tags is fine.

I think the proposal needs to be a little clearer about the matching semantics. Is the tag after the # character must be present, so a build with no tags specified doesn't match any?

I would propose:

No build tags: same semantics as present: it matches on GOOS and/or GOARCH if present, or if not, is always included.
More than one build tag: AND semantics.

This would be the same as having multiple // +build lines in a Go source file.  That would be consistent with the existing GOOS/GOARCH semantics; e.g. *_linux_amd64.syso means GOOS=linux AND GOARCH=amd64.
Where a user really wants an OR condition of two or more tags, they could add the syso file multiple times, each being a symlink with a single #tag suffix to the real syso file.
I think this latter situation will be rare: I can easily see #iphonesimulator AND #debug or #iphonesimulator AND #release.

I'm happy to update the proposal to make this more clear.

This seems very advanced and specific, and I'm a bit reluctant to make syso and file selection far more complicated just to support this rare case.
It seems like you can already make this work today. Suppose you have a package p, and you put two (or more) files like:
//go:build iphone
package p
import _ "p/iphone_syso"
and:
//go:build tv
package p
import _ "p/tv_syso"
and so on.
On iPhone, p/tv_syso won't be imported and the syso won't go to the linker. But p/iphone_syso will, and that syso will go to the linker?
Thoughts?

Yeah, this is what I suggested in https://github.com/golang/go/issues/38485#issuecomment-719962834

This seems very advanced and specific, and I'm a bit reluctant to make syso and file selection far more complicated just to support this rare case.
It seems like you can already make this work today. Suppose you have a package p, and you put two (or more) files like:
[snipped]

Thank you for reviewing the proposal, @rsc.
My outline could have been more clear on your workaround, also suggested by @cherrymui: it was referred to the "sub-package" approach without further detail.  I've updated the proposal at the top of this thread, and also included more clarity on the treatment of multiple build tags.
Compared to the workaround available today, both @cherrymui and I agree that it is not pretty.  As outlined in the update, it requires two additional Go source files and one new sub-package directory for every unique syso file.
I have several packages supporting macOS (amd64 + soon arm64), iOS (arm64), iOS simulator (amd64 + soon arm64), Android (arm64), Linux (amd64, arm64), and Windows (amd64, 386).  For each platform object, I have Release and Debug builds.  The macOS, iOS, and iOS simulator syso files can't be uniquely selected for a build without the workaround which makes the package a mess, while the other syso files remain in the package root.  This proposal would unify and simplify that layout significantly.
I appreciate the flexibility that +build and now go:build lines offer, and use the former extensively in my packages.  I would however argue that basic tag selection in both syso and Go source filenames would have been a welcome feature from the get-go.  For all Go source files where I use +build lines, I also add a decorative filename suffix to make it clear in my IDE that such files are built conditionally.  I appreciate that others might not see it that way, which is why I limited this proposal to just syso files where I believe there is no good alternative.
I'm about to start looking at a CL for this, and I expect the change to be small.  I hope you're able to keep an open mind.

Using subpackages is not pretty, but using build tags in the file name starting with "#" is also not pretty.  Subpackages, while verbose, do not require any new features and are thus clear to most Go programmers.  Build tags starting with "#" will always be a very obscure feature that seems likely to trip people up.  Given that very very few people will ever want to use this feature, and given that it can be supported without adding any new features, I personally think there is a strong argument for not adding any new features here.

Using subpackages is not pretty, but using build tags in the file name starting with "#" is also not pretty.

The lesser of the two evils, and a much better developer experience, is surely just naming one file lib_#macosx.syso over the alternative of adding a bunch of extra files and another folder just to work out a trivial limitation of the existing build constraint file selection.
The filename lib_#macosx.syso doesn't look ugly to me?  Anyone else want to chime in?

Given that very very few people will ever want to use this feature, and given that it can be supported without adding any new features, I personally think there is a strong argument for not adding any new features here.

I think more people would use this functionality if:

It becomes more widely known and/or documented.
More developers start using Go for interactive apps, where you need to link into non-Go native code much more often, and where having a "go gettable" package while statically linking that code into the executable is desirable.
The previous bullet will start becoming more commonplace as some of the Go UI projects start maturing.  That kind of work has directly led to this proposal.

The feature is rather trivial: visible only in a syso filename and nowhere else, and the implementation is also minimal as you can see in the simple CL I submitted 1/2-hour ago.
Thanks for taking the time to read the proposal, @ianlancetaylor.  I hope you'll reconsider with the above arguments in mind, and have a quick look at the associated CL.

The feature is rather trivial: visible only in a syso filename and nowhere else

Today, the *_GOOS_GOARCH naming pattern for .syso files is the same pattern used for build constraints for ordinary Go source files: there is nothing particularly special about .syso files in that regard, and anyone who has worked with platform-dependent .go files should be able to map that understanding fairly easily onto the .syso files.
I think it would be a mistake to add a separate naming convention that applies only to .syso files.
On the flip-side, users are already confused enough about the implicit build constraints for ordinary Go source files. If this new naming scheme did not apply only to .syso files, I think that would be even more confusing.
Given how rarely .syso files are used, I think that on balance it would add more complexity than it removes. The subpackage approach is already possible today and doesn't seem that bad to me, and we can always revisit in a few years if for some reason .syso usage grows dramatically.

Today, the *_GOOS_GOARCH naming pattern for .syso files is the same pattern used for build constraints for ordinary Go source files: there is nothing particularly special about .syso files in that regard, and anyone who has worked with platform-dependent .go files should be able to map that understanding fairly easily onto the .syso files.

Thanks for chiming in.  I agree with this, and the proposal doesn't change the *_GOOS_GOARCH naming pattern for those having learned using it with .go files.  It still carries across.  This proposal is additive.

I think it would be a mistake to add a separate naming convention that applies only to .syso files.

If consistency is important to you, I'd be happy to see the proposed *_#tag naming pattern extended to .go files.  (It even further simplifies the CL.)  I avoided proposing that because I expected an even louder objection than what I have seen in this thread.

On the flip-side, users are already confused enough about the implicit build constraints for ordinary Go source files. If this new naming scheme did not apply only to .syso files, I think that would be even more confusing.

I agree that developers can be easily confused about the explicit +build constraint syntax: I was pleased to see @rsc's work on improving that with the new go:build line.  I smiled when he said that people can't remember the syntax for too long because I can relate!
But the implicit build constraints in the filename are straightforward because they are so rigid and simplistic: you just summarized it with a simple string pattern in your post!  Can you point out instances where developers get confused with this?

Given how rarely .syso files are used, I think that on balance it would add more complexity than it removes. The subpackage approach is already possible today and doesn't seem that bad to me, and we can always revisit in a few years if for some reason .syso usage grows dramatically.

The prospect of living with this messy workaround for "a few years" is unappealing.  We already have syso files in over 15 packages and are just getting started.
Based on the discussion so far, what interest (if any!) is there for:

Unifying the build tag filename patterns for both .go and .syso files?
A filename pattern that is not considered "ugly"?
A new build line, like //go:syso [filename] to avoid having to use a sub-package, and with no change to the file naming pattern?

This would still require one extra .go source file to specify the above syntax and a //go:build line to constrain selection to the desired platform.
It may also require the filename to not end in .syso so it doesn't conflict with existing matching rules.

A variation on (3) like //go:syso [go:build expression] [filename] that means you can place it in any existing .go file?

Example: //go:syso macosx && amd64 lib_macosx_amd64.o, where the file lib_macosx_amd64.o sits in the same directory as the corresponding .go file.
I would place it near existing Go code that uses the resources from the syso file.

We have several Go packages that targets multiple platforms, including macOS, iOS, tvOS, iOS-simulator, tvOS-simulator, windows, linux, etc. The package has one or more syso file dependencies.
Syso files can be selected for a build by a GOOS/GOARCH filename suffix but not build tags, unlike Go source files that use // +build or //go:build lines. If we ignore tvOS for the moment, some time ago we could build a syso file for darwin/amd64 and include that in both macOS and iOS-simulator builds just fine. With recent Xcode 12, that approach no longer works:
ld: building for iOS-simulator, but linking in object file built for macOS, file '/var/folders/h7/r_43shxn71v0pq_gx72f57ph0000gn/T/go-link-172505736/000035.o' for architecture x86_64

The situation worsens when we support tvOS (+ the tvOS-simulator), and the new ARM-based Apple Silicon Macs because we can no longer use a GOOS/GOARCH filename build constraint to uniquely select the appropriate syso variant.

I wonder whether that linker error is the only problem? If so, the Go linker may be able to help by only presenting object files to the system linker that have matching LC_BUILD_VERSION load commands. The linker already tries to select an appropriate version, see for example (hostobjMachoPlatform)[https://github.com/golang/go/blob/master/src/cmd/link/internal/ld/macho.go#L1196].

@jpap It seems to me that you are arguing in favor of a change that 1) very few people will want to use; 2) you can already do today in a different, albeit more awkward, way.
A good argument here would be pointing to several different popular packages, by different people, that would use this feature if it were available.
The argument that more people are going to want to use this feature in the future doesn't seem particularly convincing to me.  It's particularly unconvincing if the reason that people want to use this feature is to bypass cgo safety in the name of speed.  If that is the problem that people are trying to address, then we should address that, not encourage increasingly complex use of a workaround.

I wonder whether that linker error is the only problem? If so, the Go linker may be able to help by only presenting object files to the system linker that have matching LC_BUILD_VERSION load commands. The linker already tries to select an appropriate version, see for example hostobjMachoPlatform.

Yes -- the Go linker could filter all of the syso files with GOOS=darwin and extract only the ones for a given platform, but in order to do that, it needs to know what the build target platform is: for example, is it iOS-device, or iOS-simulator (Apple Silicon Mac)?
In the code you've cited, it takes the first object file involved in the build, extracts the platform, and if it is macOS, inserts the required LC_VERSION_MIN_MACOSX load command, that would otherwise break lldb if missing.  That code implicitly assumes that all of the object files are built for the same target platform.  In this case, what do you do when the linker is presented with a collection of heterogeneous objects?  Declare the target platform as the one in having a majority of objects presented?  You would likely need some kind of target platform argument to make it explicit.  In the proposal here, that explicit argument is the user-defined build tag so the linker is presented only with homogenous objects.  You could condition the Go linker on such a build tag, but I don't see anyone embracing that kind of coupling.
Even if you were to use some magic in the Go linker to solve the above; you still have the problem with Android and Linux.  When you build for Android, the GOOS=linux alias will select all *_linux_arm64.syso files, for example.  ELF doesn't have an explicit platform header, so the linker won't know what these objects target, and the link will succeed so long as all import symbols are found.  Not a problem if the functions in the object are purely compute (e.g. an image processing algorithm, data compression, or hashing), because there might be no imports.  But if the object calls a platform API, you're in trouble because of the syso file selection issue.
The fundamental problem here is that GOOS was thought to be tied to a kernel, and not a "platform" or "OS".  I understand the reasons for doing so -- it simplified cross-platform support and brought Go more quickly to a bunch of new platforms; the "Go way" was also to avoid platform APIs as much as possible and hit the kernel interface directly.  And it wouldn't be a problem here, either, if we had the build tag escape hatch for these syso files.  (I recognize the workaround; but it is a mess.)
Elias, did you have have an opinion on the proposal itself?  I appreciate the thoughts going into other workarounds, but would love to know if someone else who is knee-deep in non-Go code and platform interop can see the benefits proposed here.

A good argument here would be pointing to several different popular packages, by different people, that would use this feature if it were available.

Thanks for painting a path forward.  I can't help being a little disappointed that it isn't to discuss the technical merits of the proposal, or some of the proposed alternatives, but to judge it solely on its apparent popularity, especially given how minor this feature is and how small the CL turned out to be. :)
I agree that syso usage is not an everyday use-case for the majority of Go users.  That is because of the demographic: the Go Survey clearly shows that the vast majority of Go use-cases are API/RPC services and CLIs.  Those using Go for Desktop and Mobile apps are in the tiny minority, but that is where syso shines.
This situation is chicken-and-egg: Go developers don't write GUI apps because Go lacks great UI packages.  That will change as one of more Go UI projects matures.  I feel it is a bit disingenuous to ask for references from "several different popular packages, by different people" when the number of people working in this space is miniscule.
It's great to have Elias in this thread, and I've put it to him in my last post.  His Gio project may not benefit as much from improved syso support, as his goal is to avoid interfacing with the (UI-component part of the) platform as much as possible, by rendering a completely custom UI into a blank canvas.  When you write GUI apps that interface into platform SDKs and platform UI components, you tend to rely on cgo much more frequently, and leveraging syso leads to a much better developer experience.

The argument that more people are going to want to use this feature in the future doesn't seem particularly convincing to me. It's particularly unconvincing if the reason that people want to use this feature is to bypass cgo safety in the name of speed. If that is the problem that people are trying to address, then we should address that, not encourage increasingly complex use of a workaround.

The stain of "let's use syso to bypass cgo" is unwarranted here.  Syso files are the only way to host a "Go gettable" package that uses a non-Go library that gets automatically and statically compiled into the final Go executable while using cgo as the glue.  I've already linked to a public example in a previous post.  I think this approach isn't popular because it isn't well documented.  A lot of projects that provide cgo wrappers for a non-Go library typically assume that library is pre-installed on the system.  It is a much better developer and end-user experience, and more "Go like", to have the library statically linked to the executable.
I have successfully used the statically compiled syso approach for interop with many 3rd party non-Go libraries:

Wrapping a video codec library to support image coding with H.265.
Wrapping a lossless arithmetic compression library, that has no native-Go implementation, to support interop with a popular file format.
Wrapping libpng for higher performance PNG and more control, compared to the image/png package.
Wrapping mozjpeg for higher performance and better compression over the image/jpeg package.
Wrapping littleCMS to work with ICC profiles and perform image color management.
Wrapping a disassembly library (linked above) to support static analysis of x86_64 text.
Wrapping pdfium to render PDFs into images.

I have also used the approach to statically link many 1st party non-Go libraries that interface with platform SDKs.  I am not going to list those here.  All but one of these packages are currently private; that might change in the future, but the work being done here is commercially focused.
All of these packages use cgo.

especially given how minor this feature is and how small the CL turned out to be. :)

I want to clarify that this is not a strong argument.  Every feature must be documented, maintained, and tested.  Small features are not better from a long-term maintenance perspective.  They are worse, because they are less used, less tested, and less understood, but they must be fixed when they break.
Thanks for the comments about cgo.  I wonder if this is an argument for having better interaction between go build and code that is not written in Go, so that we can build that code rather than having to ship .syso files.

especially given how minor this feature is and how small the CL turned out to be. :)

I want to clarify that this is not a strong argument. Every feature must be documented, maintained, and tested. Small features are not better from a long-term maintenance perspective. They are worse, because they are less used, less tested, and less understood, but they must be fixed when they break.

I agree on the importance of documentation, testing, and maintenance in a large, long term project.  One could argue that my CL included the first two things: it included tests, and added both godoc and go help buildconstraint usage text.  I additionally performed some manual smoke tests before pushing the CL.  Maintenance is a long-term burden on all contributors: in submitting that CL, I was essentially performing that task on the "syso" feature.  If the CL warrants further work in those areas, add it to your review, and I'd happily address it.
Broad generalizations are helpful but there are always special cases.  What you say about small features may be true in general, but in this case, having never previously looked at the relevant code, I was able to understand and produce the CL, including testing, within an hour.  While some of the Go codebase is very complex for newcomers, perhaps some parts brittle to breakage when unrelated changes are made, I feel this CL is far from both of these things. :)

Thanks for the comments about cgo. I wonder if this is an argument for having better interaction between go build and code that is not written in Go, so that we can build that code rather than having to ship .syso files.

Based on my experience, I would recommend bolstering syso support or something like that, where you are ingesting precompiled objects or archives.  The reason for this is that for many nontrivial non-Go projects, the build process is complex and sometimes bespoke.  They might use CMake, Autoconf, GYP and Ninja, etc.
One of the great things about Go is not only the ability to target across platforms, but also the ability to cross-compile easily.  Doing so with cgo is possible, but requires more care.  Doing so with cgo + a non-Go project build system is even harder.  Cross-compiling some of the 3rd party libraries I mentioned in my last post was not always an easy task.  In some sense that is what Elias is looking for in #38917 -- if you are linking prebuilt objects, there's no need to worry about SDK headers that you are licensed to only use on a certain platform. ;)
It is much easier to compartmentalize 3rd party non-Go code than work directly with its source files, especially when that code was not originally intended for consumption by Go.
In recent months I have also spent time working out how to pragmatically support cross-compilation of cross-platform apps that make extensive use of a FFI, for example, calling Objective-C APIs on Apple's platforms.  I think I'm converging on something that works really well, but some of it hinges on prebuilt objects for calls into platform SDKs to make cross-compilation a breeze.  Syso files are at the core of that, which is another reason I am so keen to see this proposal through now and not in "a few years".  (Again, this is all using cgo as the glue.)

I have a viable path to do all of that without this proposal, while still maintaining a good developer experience, but unfortunately that path involves much more work on my end and a departure from the standard Go project structure.  It would require a new build tool that sits above go build, that scans a package's imports recursively, filters prebuilt objects ("syso" files) in each package directory, while feeding the paths of those filtered objects to the external linker via -extldflags.  Since cmd/go already does this with the "syso" feature, under the limitations at the top of this thread, it makes sense to spend that hour creating a CL instead of a week or more effort on a new non-standard tool and live on a non-standard project layout.
If there's any interest in discussing some of my earlier suggestions, I'd be all for it.  Otherwise I fear we're all wasting our time here.  For that, I apologize, no hard feelings, and thanks for the discussion. I can't wait for you to see what I've been working on...

I wonder whether that linker error is the only problem? If so, the Go linker may be able to help by only presenting object files to the system linker that have matching LC_BUILD_VERSION load commands. The linker already tries to select an appropriate version, see for example hostobjMachoPlatform.

Yes -- the Go linker could filter all of the syso files with GOOS=darwin and extract only the ones for a given platform, but in order to do that, it needs to know what the build target platform is: for example, is it iOS-device, or iOS-simulator (Apple Silicon Mac)?
In the code you've cited, it takes the first object file involved in the build, extracts the platform, and if it is macOS, inserts the required LC_VERSION_MIN_MACOSX load command, that would otherwise break lldb if missing. That code implicitly assumes that all of the object files are built for the same target platform. In this case, what do you do when the linker is presented with a collection of heterogeneous objects? Declare the target platform as the one having a majority of objects presented? You would likely need some kind of target platform argument to make it explicit. In the proposal here, that explicit argument is the user-defined build tag so the linker is presented only with homogenous objects. You could condition the Go linker on such a build tag, but I don't see anyone embracing that kind of coupling.

Build tags in filenames is a blunt instrument. I believe there are lighter weight options available. Off the top of my head:

Improve the LC_VERSION_MIN_ detection. For example, the cgo tool or the Go linker could query $CC for the platform, and then ignore incompatible syso objects.
If automatic detection is infeasible, add a linker argument specifying the platform variant (tvOS/iOS/watchOS), just like the existing -H windowsgui flag. Then, ignore incompatible sys objects.

Even if you were to use some magic in the Go linker to solve the above; you still have the problem with Android and Linux. When you build for Android, the GOOS=linux alias will select all *_linux_arm64.syso files, for example. ELF doesn't have an explicit platform header, so the linker won't know what these objects target, and the link will succeed so long as all import symbols are found. Not a problem if the functions in the object are purely compute (e.g. an image processing algorithm, data compression, or hashing), because there might be no imports. But if the object calls a platform API, you're in trouble because of the syso file selection issue.

It's true that Android support for the above scheme seems harder. Without knowing more about your particular setup, I don't have any recommendations other than an explicit linker flag.

The fundamental problem here is that GOOS was thought to be tied to a kernel, and not a "platform" or "OS". I understand the reasons for doing so -- it simplified cross-platform support and brought Go more quickly to a bunch of new platforms; the "Go way" was also to avoid platform APIs as much as possible and hit the kernel interface directly. And it wouldn't be a problem here, either, if we had the build tag escape hatch for these syso files. (I recognize the workaround; but it is a mess.)
Elias, did you have have an opinion on the proposal itself? I appreciate the thoughts going into other workarounds, but would love to know if someone else who is knee-deep in non-Go code and platform interop can see the benefits proposed here.

I would love easy linking of static libraries. I'll probably need it for integrating something like Harfbuzz in Gio.
However, this proposal adds public facing machinery that's not clear is necessary, in particular when there's a (messy) workaround. For something as obscure as syso files, I believe there is value in searching for better solutions before resorting to build tags in filenames. There's ample time; the next freeze is 5 months away.

Appreciate your reply, Elias.  Some comments inline below.

Build tags in filenames is a blunt instrument. I believe there are lighter weight options available. Off the top of my head:

Improve the LC_VERSION_MIN_ detection. For example, the cgo tool or the Go linker could query $CC for the platform, and then ignore incompatible syso objects.

You could invoke the compiler to produce a dummy object, then extract the platform from it.  The extra round-trip would also elongate build times.

If automatic detection is infeasible, add a linker argument specifying the platform variant (tvOS/iOS/watchOS), just like the existing -H windowsgui flag. Then, ignore incompatible sys objects.

That is possible, however how can you deal with Debug vs. Release object selection?  Build tags offer a lot more flexibility because their meaning is user-defined, and not hard-coded into a flag.

It's true that Android support for the above scheme seems harder. Without knowing more about your particular setup, I don't have any recommendations other than an explicit linker flag.

What information would you like to know about my setup?  Based on an explicit linker flag, and the inability to discern Android ELF from "Desktop Linux" ELF, how can we filter syso files in this case?

However, this proposal adds public facing machinery that's not clear is necessary, in particular when there's a (messy) workaround. For something as obscure as syso files, I believe there is value in searching for better solutions before resorting to build tags in filenames. There's ample time; the next freeze is 5 months away.

I look forward to your suggestions, and any feedback you might have on my other alternative proposals.

If automatic detection is infeasible, add a linker argument specifying the platform variant (tvOS/iOS/watchOS), just like the existing -H windowsgui flag. Then, ignore incompatible sys objects.

That is possible, however how can you deal with Debug vs. Release object selection? Build tags offer a lot more flexibility because their meaning is user-defined, and not hard-coded into a flag.

Having separate debug vs release builds is unusual in Go, so I'm not sure how far the Go tools should be stretched to accommodate that variant axis. The ugly workaround seems good enough here.

It's true that Android support for the above scheme seems harder. Without knowing more about your particular setup, I don't have any recommendations other than an explicit linker flag.

What information would you like to know about my setup? Based on an explicit linker flag, and the inability to discern Android ELF from "Desktop Linux" ELF, how can we filter syso files in this case?

"How to filter syso files" smells like an XY problem to me. Why do you need different syso files for Android vs Linux? Can the static libraries be built in a way they don't refer to Android-specific APIs?

However, this proposal adds public facing machinery that's not clear is necessary, in particular when there's a (messy) workaround. For something as obscure as syso files, I believe there is value in searching for better solutions before resorting to build tags in filenames. There's ample time; the next freeze is 5 months away.

I look forward to your suggestions, and any feedback you might have on my other alternative proposals.

If we decide filtering syso files is required, then I like your magic comments better. Perhaps add them to #cgo directives, because they understand build constraints already:
// #cgo android,amd64 syso: blah_android_amd64.syso

Having separate debug vs release builds is unusual in Go, so I'm not sure how far the Go tools should be stretched to accommodate that variant axis. The ugly workaround seems good enough here.

Go doesn't suffer from the debug-vs-release dichotomy because it doesn't perform extensive optimizations, favoring faster build times (and a simpler compiler?).  There's also good reason for keeping debugging information in the binary for panic stack traces at runtime.  These things are fine if you're deploying a backend service on your own/rented network; but for user interactive apps that run on foreign machines it's a bit different:

Debug features to introspect/control the UI state remotely becomes important for builds that run on an attached device (e.g. iPhone or Android phone connected over USB).  You don't want some kind of UI debug server still embedded and active in an app released to an end-user.  There are some aspects to this that you absolutely must not keep in the executable, for example calls to mmap to create code pages that flip between {w, rx}-states at runtime to support hot code reload.  Leaving them in would certainly leave you vulnerable to being rejected from the Apple's App Store, for instance.

Statically linked non-Go code (using cgo!!) built using -Os is a good idea for energy efficiency on mobile devices and smaller executables in general.  But debugging those builds with lldb becomes a pain, so an -O0 debug build becomes desirable.  Crash reporting of such code is largely a solved problem with exception handling "crash reporters" and remote symbolication. (Having a similar thing on the Go side is a topic for another GitHub issue.)

I am really looking forward to gollvm maturing so we can get the best of both worlds: fast compilation during development with gc, and a more extensive optimization for release builds with gollvm, which will also produce LLVM bitcode that is encouraged (read: perhaps required in future) by Apple on their App Store submissions.

"How to filter syso files" smells like an XY problem to me. Why do you need different syso files for Android vs Linux? Can the static libraries be built in a way they don't refer to Android-specific APIs?

We need native code that uses the C(++) ABI on:

Android to hit the JNI to interact with the ART, and/or to hit the NDK.
Linux to hit a UI toolkit like GTK, Tk, Qt, etc.

That native code is most easily written in C(++), interfaced to Go via cgo (+ SWIG, or an extern C wrapper).
If you use {.c, .cc} files in your Go project and/or self-contained cgo, then yes, you don't need platform-specific syso files anymore, but now cross compilation becomes more difficult, especially from macOS and Windows.  That, again, relates to your shameless plug. When that native code sits in a syso file, cross-compiling of the Go project becomes a breeze, and we're now back on-topic with this proposal.
I see X == "need native code that hits platform-specific APIs" and Y == "need syso".

If we decide filtering syso files is required, then I like your magic comments better. Perhaps add them to #cgo directive, because it understands build constraints already:
// #cgo android,amd64 syso: blah_android_amd64.syso

I would caution against associating syso with cgo here: they are independent features, even though they might work well together under many circumstances.

FWIW, I appreciate that it's a little more work in this use case, but sometimes that's preferable to complicating the general case for something that most people will never see. It becomes one more thing people have to understand when they do see it. If you know about regular build tags and subdirectories (which most people do), then you know what you're looking at when you see the subdirectory solution. On the other hand, if you see a file named lib_#appletvsimulator_amd64.syso, that's surprising and uninterpretable without going off to learn a new rule.

Based on the discussion above, this seems like a likely decline.

Given the current state of this proposal, I've instead implemented a custom build tool with a nonstandard Go project layout in favor of a better DX over the proposed workaround.
Thank you for the consideration and detailed discussion by all.

This seems very advanced and specific, and I'm a bit reluctant to make syso and file selection far more complicated just to support this rare case.
It seems like you can already make this work today. Suppose you have a package p, and you put two (or more) files like:
//go:build iphone
package p
import _ "p/iphone_syso"

and:
//go:build tv
package p
import _ "p/tv_syso"

and so on.
On iPhone, p/tv_syso won't be imported and the syso won't go to the linker. But p/iphone_syso will, and that syso will go to the linker?
Thoughts?

A rather unfortunate side effect of this is that you'll need to instruct the linker to look up symbols at runtime. So, this program that previously wouldn't compile now crashes: https://play.golang.org/p/zLi2iNfwiZO

A rather unfortunate side effect of using multiple packages is that you'll need to instruct the linker to look up symbols at runtime. So, this program that previously wouldn't compile now crashes: https://play.golang.org/p/zLi2iNfwiZO
This is a problem I've run into when creating .syso files for x86-64 Android and Linux. Since the android GOOS also matches the linux GOOS, the compiler attempts to use both lib_android_amd64.syso and lib_linux_amd64.syso when GOOS=android.
@rsc I'd like to reopen this issue (or make another if you'd prefer, I'm not a fan of the "#" syntax). .syso files are a little esoteric, but incredibly useful.

As far as I can tell you can address that issue using the approach described at https://github.com/golang/go/issues/42477#issuecomment-725591875.  If not, why not?

@ianlancetaylor yes, multiple packages works. But using a tennis racket as a canoe paddle also works. :)
In particular, cmd/cgo does not pull in other packages, so linking fails unless you instruct the linker to ignore undefined symbols. This could cause the program to explode at runtime instead of compile time.
Or, you have to manually link against the .syso files in #cgo directives, which mostly defeats the purpose of using .syso files.

I think we discussed all the tradeoffs above, and came to a decision.  If multiple packages works, then I don't see a need to reconsider.  I understand that this is more work, but it is possible, and it's for an unusual case.  Thanks.


Add raw output to invoke-restmethod for when you don't want any parsing.

Still missing a unittest. but rest seems pretty simple.

@paulgmiller,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

Hmm in manually testing seems to take -rawoutput but I don't see it in the help.

Trying to understand the use case of why someone would use this instead of just invoke-webrequest if they want to parse it themselves?

Yeah, this seems like redundant behavior on top of Invoke-WebRequest to me. Both HtmlWebResponseObject (in Windows PowerShell) and BasicHtmlWebResponseObject (in PowerShell Core) contain the RawContent property, and Invoke-WebRequest supports all the same web request methods.
@paulgmiller Do you have a scenario where Invoke-WebRequest doesn't fulfill your needs?

Waiting on a response to:

@paulgmiller Do you have a scenario where Invoke-WebRequest doesn't fulfill your needs?

Closing because of no response.  Please re-open with an answer.  Thanks


Change the output of validation tools to an object

Change the output to a validation object so that Json serialization can handle it better.

alluxio-bot, merge this please

alluxio-bot, cherry-pick this to branch-2.4 please

alluxio-bot, cherry-pick this to branch-2.4 please


electrs: 0.9.5 -> 0.9.6

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@prusnak @erikarvstedt
GPG verify hash with
#!/usr/bin/env nix-shell
#!nix-shell -i bash -p git gnupg nix_2_4
set -euo pipefail

newVersion=0.9.6

# Fetch release and GPG-verify the content hash
tmpdir=$(mktemp -d /tmp/electrs-verify-gpg.XXX)
repo=$tmpdir/repo
trap "rm -rf $tmpdir" EXIT
git clone --depth 1 --branch v${newVersion} -c advice.detachedHead=false https://github.com/romanz/electrs $repo
export GNUPGHOME=$tmpdir
# Fetch romanz's key
gpg --keyserver hkps://keys.openpgp.org --recv-keys 15c8c3574ae4f1e25f3f35c587cae5fa46917cbb 2> /dev/null
echo
echo "Verifying commit"
git -C $repo verify-tag v${newVersion}
rm -rf $repo/.git
newHash=$(nix hash path $repo)
rm -rf $tmpdir
echo $newHash

Tested with nix-bitcoin VM test

@GrahamcOfBorg build electrs


fzf bash history CTRL+R not working

Category

[ ] fzf binary
[ ] fzf-tmux script
[ ] Key bindings
[ ] Completion
[ ] Vim
[ ] Neovim
[* ] Etc.

OS

[* ] Linux
[ ] Mac OS X
[ ] Windows
[ ] Windows Subsystem for Linux
[ ] Etc.

Shell

[* ] bash
[ ] zsh
[ ] fish

I used to be able to invoke an fzf search with CTRL+R in bash.  However, last night I killed my home dir by accident.  I had to rebuild.  However, now fzf CTRL+R doesn't work, all I get is the standard history search.  My bashrc does include
[ -f ~/.fzf.bash ] && source ~/.fzf.bash
Originally, I think I had installed fzf through a vim plugin.  But I'm not understanding what the problem is now.  Anyway, the fzf history search was very convenient, I would like to get it back.

Rerun the install script.

You mean from vim? Could you point me in the direction of the install
script?
Mindaugas
ミンダウガス
On Dec 31, 2017 20:55, "Junegunn Choi"<EMAIL_ADDRESS>wrote:

Rerun the install script.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/junegunn/fzf/issues/1190#issuecomment-354599909, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AWA_Mp1PTKpikAqvkj8LwDiRMUywFb_Cks5tF3YWgaJpZM4RP0Mi
.

https://github.com/junegunn/fzf#installation

Thank you
Mindaugas
ミンダウガス
On Dec 31, 2017 21:04, "Junegunn Choi"<EMAIL_ADDRESS>wrote:

https://github.com/junegunn/fzf#installation
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/junegunn/fzf/issues/1190#issuecomment-354600315, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AWA_Mn4-uKbVAO64gD684T5dwmdKOORKks5tF3hogaJpZM4RP0Mi
.

worth noting that my macOS bash environment didn't come with .bashrc, but rather .bash_profile. So my CTRL-R didn't work until I merged the two, by sourcing .bashrc from .bash_profile as discussed here: https://apple.stackexchange.com/questions/51036/what-is-the-difference-between-bash-profile-and-bashrc#comment417289_51038

Hello.
I've missed the moment when it stopped working. Tried to run ./install again but no success.
Now
Ubuntu: Ctrl+R doesn't work, Ctrl+t works
Arch: neither of two works
This is what I have in my .zshrc:
if [[ -f "$HOME/.fzf.zsh" ]]; then
  source ~/.fzf.zsh
  export FZF_DEFAULT_COMMAND='fd --type file --follow --hidden --exclude .git'
  export FZF_DEFAULT_OPTS="--height 60% --layout=reverse"
  export FZF_CTRL_T_COMMAND="$FZF_DEFAULT_COMMAND"
fi

When launching fzf manually everything works great.
Could you pls help me with this?

I run Ubuntu
This command enabled bash Default bindings
Enable Bash Key Bindings with CNTR+R [مهم]```
fzf keybindings for Bash:
source /usr/share/doc/fzf/examples/key-bindings.bash

In my case,
source /usr/local/share/zsh-vi-mode/zsh-vi-mode.zsh

in ~/.zshrc was blocking CTRL+R


📃: Rock paper Scissors Game

:red_circle: Title : Rock Paper Scissors Game
:red_circle: Aim : To understand JavaScript fundamentals
:red_circle: Tech Stack Used : Html, CSS and JavaScript
Screenshots 📷

:white_check_mark: To be Mentioned while taking the issue :

Full name : G AVINESHWAR
What is your participant role? (Mention the Open Source Program name. Eg. GSSOC, SSOC, JWOC, etc.)
GSSOC

Happy Contributing 🚀
All the best. Enjoy your open source journey ahead. 😎

no games allowed