E2E test for tokens only namespaces

Adds an E2E test for namespaces with only a tokens plugin
Moves keynormalization config under namespaces.predefined[].assets.manager.keynormalization
Adds common for code adding namespaces, resetting firefly, etc. to a common package

Codecov Report

Merging #919 (39a8662) into main (3ae7ba8) will increase coverage by 0.01%.
The diff coverage is 100.00%.

:exclamation: Current head 39a8662 differs from pull request most recent head 502894a. Consider uploading reports for the commit 502894a to get more accurate results

@@            Coverage Diff             @@
##             main     #919      +/-   ##
==========================================
+ Coverage   99.97%   99.98%   +0.01%
==========================================
  Files         299      299
  Lines       19475    19481       +6
==========================================
+ Hits        19470    19478       +8
+ Misses          4        3       -1
+ Partials        1        0       -1

Impacted Files
Coverage Δ

internal/coreconfig/coreconfig.go
100.00% <ø> (ø)

internal/assets/manager.go
100.00% <100.00%> (ø)

internal/namespace/config.go
100.00% <100.00%> (ø)

internal/namespace/manager.go
100.00% <100.00%> (ø)

internal/orchestrator/orchestrator.go
100.00% <100.00%> (ø)

internal/tokens/fftokens/fftokens.go
100.00% <0.00%> (+0.46%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Adding the migration_consideration label because asset.manager.keynormalization config has been moved under namespaces.predefined[].asset.manager.keynormalization


Roll Engine from e4c4a2f2052f to 4c5f30c1fb47 (3 revisions)

https://github.com/flutter/engine/compare/e4c4a2f2052f...4c5f30c1fb47
2021-02-02<EMAIL_ADDRESS>Roll Skia from 931cad1d74bb to e9e8e5d771c5 (8 revisions) (flutter/engine#24138)
2021-02-02<EMAIL_ADDRESS>Avoid calling Dart timeline APIs during Dart_Cleanup (flutter/engine#24007)
2021-02-02<EMAIL_ADDRESS>drawBitmap is deprecated, use drawImage (flutter/engine#24135)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/master/autoroll/README.md

Commit queue failed; closing this roll.


Bug fix for filter command failing with inputs

PR aims to fix the bug identified by PE-D :
Filter command fails when user enters valid inputs in lowercase (#147)

Expected behaviour : Filter should work regardless of case and fetch the relevant internship contacts

Actual behaviour : Filter command does not function when there are contacts with lowercase tags & there is no workaround available as with current implementation of filter - unable to successfully locate the tags with lowercase

Remedy / Bug Fix : ensure that Tags.getTagShort() returns as UpperCase to avoid such inaccuracies.

bug fixed #174


AutoLogon mode not working when installed using Azure Extension or DevTestLabs artifact

Agent version and platform
2.124.0 (on Windows 10 Pro x64)
VSTS
What's not working?
I install the agent using an Azure Extension or DevTestLabs Artifact while deploying a VM. But then the AutoLogon mode does not work. This means that when the machine is started, the agent is not automatically started. I use the same user-account to run the agent as to logon to the machine. When I run the same command when logged into the machine (RDP), it does work. By comparing the logs I think it has something to do with the way the user is logged in when running the command from the Azure Extension or DevTestLabs Artifact (see logging).
I see this behaviour on Windows 10 Pro v.1709 (x64) and Windows Server 2016 (x64). What is strange is that on Windows 10 RS2 Pro (x64), it does work as expected when installed from Azure Extension or DevTestLabs artifact (I tried this as an experiment).
This is the command-line I use (secrets obfuscated):
 .\config.cmd --unattended --url https://myaccount.visualstudio.com --auth pat --token 61...8a --pool MyPool --runAsAutoLogon --windowsLogonAccount testuser1 --windowsLogonPassword 0a...4R --overwriteAutoLogon
Agent and Worker's diag log
In the logs I find a difference. When running from the Azure Extension or DevTestLabs Artifact, a warning is emitted in the log about not being able to find the user's SID in the registry for inserting the run task:
From Azure Extension or DevTestLabs Artifact (notice the warning):
...
[2017-11-03 11:09:45Z INFO HostContext] Well known directory 'Root': 'C:\Build'
[2017-11-03 11:09:45Z INFO AutoLogonRegistryManager] Agent auto logon startup command: 'C:\Windows\system32\cmd.exe /D /S /C start "Agent with AutoLogon" "C:\Build\run.cmd" --startuptype autostartup'
-> [2017-11-03 11:09:45Z WARN WindowsRegistryManager] Couldnt get the subkey 'S-1-5-21-2773755948-758306923-2095025604-500\SOFTWARE\Microsoft\Windows\CurrentVersion\Run. Will not be able to set the value.
[2017-11-03 11:09:45Z INFO NativeWindowsServiceHelper] Successfully unloaded the profile for testuser1.
[2017-11-03 11:09:45Z INFO WhichUtil] Which: 'powercfg.exe'
[2017-11-03 11:09:45Z INFO WhichUtil] Location: 'C:\Windows\system32\powercfg.exe'
[2017-11-03 11:09:45Z INFO AutoLogonManager] Running powercfg.exe with /Change monitor-timeout-ac 0
[2017-11-03 11:09:45Z INFO ProcessInvoker] Starting process:
...

When logged into the machine:
...
[2017-11-03 11:26:53Z INFO HostContext] Well known directory 'Root': 'C:\Build'
[2017-11-03 11:26:53Z INFO AutoLogonRegistryManager] Agent auto logon startup command: 'C:\Windows\system32\cmd.exe /D /S /C start "Agent with AutoLogon" "C:\Build\run.cmd" --startuptype autostartup'
[2017-11-03 11:26:53Z INFO WhichUtil] Which: 'powercfg.exe'
[2017-11-03 11:26:53Z INFO WhichUtil] Location: 'C:\Windows\system32\powercfg.exe'
[2017-11-03 11:26:53Z INFO AutoLogonManager] Running powercfg.exe with /Change monitor-timeout-ac 0
[2017-11-03 11:26:53Z INFO ProcessInvoker] Starting process:
...

@dpksinghal @nigurr for auto-logon question.

@EdwinVW Thanks for reaching out. We will be investigating into this issue. Can you please try using RS2 build to unblock temporarily?

Yes, I'm not blocked at the moment. Thanks for looking into this issue!

@EdwinVW sorry for the delayed response. Can you please share the template which you are using to spin up the agent?
or Sample build definition?

@nigurr I don't use a template or build definition. I use a DevTestLabs artifact (which is in fact a Powershell script) to install the agent. I've included the PowerShell script I used as attachment.
install-buildagent.zip

@EdwinVW from the DevTestlabs repo, there's no official support for interactive agent - https://github.com/Azure/azure-devtestlab/blob/master/Artifacts/windows-vsts-build-agent/vsts-agent-install.ps1#L248
Can you please check once?

@nigurr I know. The artifact I mentioned in my last comment was created by me. I took the standard artifact you're referring to and changed the command-line in this script so it uses uses the autologon feature. I've attached the script in my last comment.

Could you try a few things:

Have you tried re-running the artifact after say manually running the script on the vm once via rdp
Manually check if the subkey exists (the one the script claims doesn't)
Try out a simple powershell script that just tries to read this subkey and run it as an artifact or extension

It is a possibility that when running scripts via an atifact they may not have access to certain parts of the registry on those particular OSes
This is just a hunch but it will provide us with more info as to what is going on.

When we join the Machine to the Domain the Autologon does not work !
Was this fixed? Using

Agent.Version
2.171.1


CompCor

It seems my original issue regarding CompCorAuto was closed. I have an follow up question here regarding the output of CompCor. I am trying to extract the time series of csf and white matter from my functional mri data. The goal of doing that is to regress those two confounds out before doing other analyses. With help from the ANTS team, now I can extract the time series, using the following command, but my question is, is there a way to set the output to be the cleaned fmri data, that is, fmri data that had the csf and white matter time series extracted out already. With the codes below, the output is the time series of the mask, either it is a csf or a white matter mask. Thanks a lot!
img<-antsImageRead("fm_run1_al.nii")
maskImg<-antsImageRead("anat_3.5.nii")
mask<-getMask(maskImg)
mat<-timeseries2matrix(img, mask)
compcorrdf<-compcor(mat)

you might want to use preprocessfMRI ... type ?preprocessfMRI in R and read the help.   alternatively, see code below.
avgbold = getAverageOfTimeSeries( bold )
mk  = getMask( avgbold )
mat = timeseries2matrix( bold, mk )
cc = compcor( mat )
rmat=residuals( lm( mat ~ cc ) )
rimg=matrix2timeseries( bold, mk, rmat )

just note: compcor does not "  extract the time series of csf and white matter from ... functional mri data "
instead, it extracts the principal vectors from a timeseries matrix derived from high-variance voxels.  this may or may not be similar to csf and white matter signal.
from example above, you can do
plot( ts(cc) )

to look at the components.


WIP: Support Firefox web extenstions

Firefox now has support for a web extensions API that mimics what's in Chrome.
With Firefox Quantum coming out in November I was curious to see if Marinara works in Firefox now, and surprise it does! I'm still testing so I'm sure I'll find more issues but this is enough to get it installed and running without any errors. Will use it for a few days to see how it goes, but any feedback now is welcome.

This is fantastic! I had been reading about Firefox/Chrome API compatibility, so I was wondering the same thing. I'm impressed by the FF team that the extension works so well as-is.
I have to get setup with Firefox to publish, but this is a great start. Thanks for digging into this.

I found a couple other issues as I've been using this. In particular, the notifications aren't working properly when the timer runs out. I'll send you another PR but could use some feedback on how to do the feature detection or fallback. Don't want to make the codepaths too ugly. :P

Awesome, thanks. The notification code is rough already, so don't worry too much about it. We can always clean it up.
If it gets too complicated, I can picture a polymorphic control object that provides a high-level API like "show notification" that is implemented differently for Chrome/FF. That way we can avoid if/else try/catch everywhere.
Also, I have a small seed of an idea to eventually port this to Electron for desktop support, so long-term there will almost certainly be a control layer on top of the different platforms anyway.


Greek Yot (U+037F) - Invalid character. ts(1127)

The character GREEK CAPITAL LETTER YOT (U+037F) causes the error Invalid character. ts(1127). No other Greek characters cause this. See this repo if you need to see more.
No other characters from the Greek/Coptic block cause this issue. I cannot find any invalid characters in my code that would be causing this issue.
TypeScript Version:  3.8.3 and tested with 4.0.0-dev.20200507
Search Terms: Invalid character, ts(1127)
Code
// src/grcCharsTrans.ts
type Dict = { [key: string]: string };
export const transliterateMap: Dict = {
  Α: "a", // GREEK CAPITAL LETTER ALPHA (U+0391)
  Β: "b", // GREEK CAPITAL LETTER BETA (U+0392)
  ...
  Ϳ: "", // GREEK CAPITAL LETTER YOT (U+037F)
}

Related Issues: https://github.com/Microsoft/TypeScript/issues/8882
VSCode: 1.34.0 (yes, this is old; this is the highest version my os can support) Perhaps a VSCode issue?
fyi, this is my first time submitting an issue. Apologies if any pertinent information is missing!

This doesn't seem to repro in the playground or locally with the repo you provided. I do think there must have been some sort of problem elsewhere, maybe with how VS Code was sending bytes across the wire previously (UTF-8 vs. UTF-16 maybe?). Maybe @mjbvz knows more.
(Which OS are you on?)

OSX 10.9.5

@charlesLoder I also can't repro this using up to date versions of VS Code. Please let us know if you see this on VS Code 1.45+


[Mention The Feature]:  Add Login and Sign-Up Pages to the Website

Is your feature request related to a problem? Please describe.
Integrating a dedicated login and sign-up page will enhance user experience by providing a secure and easy way to manage accounts. This will streamline user authentication, allowing returning users to quickly log in and new users to register seamlessly. Additionally, it will enable better personalization of the website and increase user engagement by offering features like password recovery and social media login options.
Kindly assign this issue to me under GSSoC-Extd and Hacktoberfest.
Describe the solution you'd like.
No response
Describe alternatives you've considered.
No response
Additional context.
No response
Show us the magic with screenshots
No response
Checklist

[X] I have checked the Existing Issues
[X] I have read the Contributing Guidelines
[X] I want to work on this issue. (optional)

we will look for the login and sign up page for later, because there were same issues raised before, but didn't assigned till now.
For Now i am closing this issue.
Please understand.
But don't worry you can look for other issues.
And please do continue to work on other projects.


Using PSGallery to Install Pester in AppVeyor File

This is for PowerShell/DscResources#147.

This change is

Type cast error causing test failures submitted in #8.


Disable ppuProtocolVersionL in Conway

Conway will have a new mechanism for updating the protocol version, namely the HardFork govaction.
However, we should not remove it from PParams, because there are a lot of places where we rely on the ProtVer being there.
This means that there is a trick needed to disable in PParams HKD for PParamsUpdate, but not PParams

Resolved by #3766


Fail to configure gcp region on ubuntu 20.04 python 3.8

With the following error:
(3) Configuring GCP:
    Do you want to configure GCP support in Skyplane? [Y/n]: Y
    GCP credentials found in GCP CLI
    GCP credentials found, do you want to enable GCP support in Skyplane? [Y/n]: Y
    Enter the GCP project ID [flash-ward-346914]:
    Using GCP service account skyplane-manual
    Error saving GCP region config
    <HttpError 403 when requesting https://cloudresourcemanager.googleapis.com/v1/projects/flash-ward-346914:setIamPolicy?alt=json returned "Policy update access denied.". Details: "Policy update access denied.">
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/cli/cli_impl/init.py", line 251, in load_gcp_config
    auth.save_region_config()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/utils/imports.py", line 33, in wrapped
    return fn(*modules_imported, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/compute/gcp/gcp_auth.py", line 32, in save_region_config
    service_account_credentials_file = self.service_account_credentials  # force creation of file
  File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/compute/gcp/gcp_auth.py", line 68, in service_account_credentials
    self._service_account_email = self.create_service_account(self.service_account_name)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/compute/gcp/gcp_auth.py", line 179, in create_service_account
    return retry_backoff(read_modify_write)  # retry loop needed for concurrent policy modifications
  File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/utils/retry.py", line 30, in retry_backoff
    raise e
  File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/utils/retry.py", line 27, in retry_backoff
    return fn()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/compute/gcp/gcp_auth.py", line 176, in read_modify_write
    service.projects().setIamPolicy(resource=self.project_id, body={"policy": policy}).execute()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://cloudresourcemanager.googleapis.com/v1/projects/flash-ward-346914:setIamPolicy?alt=json returned "Policy update access denied.". Details: "Policy update access denied.">

    Disabling Google Cloud support

Thanks for posting this @dbickson! Do the GCP CLI credentials you're using have permissions to create service accounts? Unfortunately, we have to create a Skyplane service account to use the Google XML API.
If your GCP CLI credentials don't have sufficient permissions, do you have permissions to manually create a service account from the GCP console? You can create service accounts from "IAM & Admin" -> "Service Accounts". If that works,  can send more instructions for how to connect Skyplane with an existing service account.

Hi @sarahwooders when looking at IAM-> service accounts in the gc console I see

<EMAIL_ADDRESS>
skyplane-manual

The ec2 region error was ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │ │ │      api_params = {}                                                                         │ │ │ │     error_class = <class 'botocore.exceptions.ClientError'>                                  │ │ │ │      error_code = 'UnauthorizedOperation'                                                    │ │ │ │  event_response = None                                                                       │ │ │ │         handler = <function inject_api_version_header_if_needed at 0x7f973c1dd040>           │ │ │ │            http = <botocore.awsrequest.AWSResponse object at 0x7f973b685c10>                 │ │ │ │ operation_model = OperationModel(name=DescribeRegions)                                       │ │ │ │  operation_name = 'DescribeRegions'                                                          │ │ │ │ parsed_response = {                                                                          │ │ │ │                   │   'Error': {                                                             │ │ │ │                   │   │   'Code': 'UnauthorizedOperation',                                   │ │ │ │                   │   │   'Message': 'You are not authorized to perform this operation.'     │ │ │ │                   │   },                                                                     │ │ │ │                   │   'ResponseMetadata': {                                                  │ │ │ │                   │   │   'RequestId': '9af51a6b-ac65-4fdc-92e0-edde962c55b6',               │ │ │ │                   │   │   'HTTPStatusCode': 403,                                             │ │ │ │                   │   │   'HTTPHeaders': {                                                   │ │ │ │                   │   │   │   'x-amzn-requestid': '9af51a6b-ac65-4fdc-92e0-edde962c55b6',    │ │ │ │                   │   │   │   'cache-control': 'no-cache, no-store',                         │ │ │ │                   │   │   │   'strict-transport-security': 'max-age=31536000;                │ │ │ │                   includeSubDomains',                                                        │ │ │ │                   │   │   │   'vary': 'accept-encoding',                                     │ │ │ │                   │   │   │   'content-type': 'text/xml;charset=UTF-8',                      │ │ │ │                   │   │   │   'transfer-encoding': 'chunked',                                │ │ │ │                   │   │   │   'date': 'Tue, 11 Oct 2022 04:26:50 GMT',                       │ │ │ │                   │   │   │   'server': 'AmazonEC2'                                          │ │ │ │                   │   │   },                                                                 │ │ │ │                   │   │   'RetryAttempts': 0                                                 │ │ │ │                   │   }                                                                      │ │ │ │                   }                                                                          │ │ │ │ request_context = {                                                                          │ │ │ │                   │   'client_region': 'us-east-1',                                          │ │ │ │                   │   'client_config': <botocore.config.Config object at 0x7f973b6d1a90>,    │ │ │ │                   │   'has_streaming_input': False,                                          │ │ │ │                   │   'auth_type': None,                                                     │ │ │ │                   │   'retries': {                                                           │ │ │ │                   │   │   'attempt': 1,                                                      │ │ │ │                   │   │   'invocation-id': 'b37d8858-c332-4b57-9290-f5fc90d11122',           │ │ │ │                   │   │   'max': 5                                                           │ │ │ │                   │   },                                                                     │ │ │ │                   │   'timestamp': '20221011T042650Z'                                        │ │ │ │                   }                                                                          │ │ │ │    request_dict = {                                                                          │ │ │ │                   │   'url_path': '/',                                                       │ │ │ │                   │   'query_string': '',                                                    │ │ │ │                   │   'method': 'POST',                                                      │ │ │ │                   │   'headers': {                                                           │ │ │ │                   │   │   'Content-Type': 'application/x-www-form-urlencoded;                │ │ │ │                   charset=utf-8',                                                            │ │ │ │                   │   │   'User-Agent': 'Boto3/1.24.89 Python/3.8.10 Linux/5.15.0-1019-aws   │ │ │ │                   Botocore/1.27.89'                                                          │ │ │ │                   │   },                                                                     │ │ │ │                   │   'body': {'Action': 'DescribeRegions', 'Version': '2016-11-15'},        │ │ │ │                   │   'url': 'https://ec2.us-east-1.amazonaws.com/',                         │ │ │ │                   │   'context': {                                                           │ │ │ │                   │   │   'client_region': 'us-east-1',                                      │ │ │ │                   │   │   'client_config': <botocore.config.Config object at                 │ │ │ │                   0x7f973b6d1a90>,                                                           │ │ │ │                   │   │   'has_streaming_input': False,                                      │ │ │ │                   │   │   'auth_type': None,                                                 │ │ │ │                   │   │   'retries': {                                                       │ │ │ │                   │   │   │   'attempt': 1,                                                  │ │ │ │                   │   │   │   'invocation-id': 'b37d8858-c332-4b57-9290-f5fc90d11122',       │ │ │ │                   │   │   │   'max': 5                                                       │ │ │ │                   │   │   },                                                                 │ │ │ │                   │   │   'timestamp': '20221011T042650Z'                                    │ │ │ │                   │   }                                                                      │ │ │ │                   }                                                                          │ │ │ │            self = <botocore.client.EC2 object at 0x7f973b6d19a0>                             │ │ │ │      service_id = 'ec2'                                                                      │ │ │ │    service_name = 'ec2'                                                                      │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

Could you please try running gcloud auth list, and seeing what account is active? Then, for the active account, could you check the role that account (under "principal") has in the "IAM & Admin" -> "IAM" tab? For example, my active account is<EMAIL_ADDRESS>which is shown as the owner:

Just wanted to check this since the error message is implying your GCP account doesn't have permissions to create a service account.

For your AWS error, could you please check if the command aws ec2 describe-regions works for you? If that gets the same error, you may need to modify the policy attached to the IAM account you are using (as described here https://stackoverflow.com/questions/28222445/aws-cli-client-unauthorizedoperation-even-when-keys-are-set).

HI @sarahwooders thanks for the helpful feedback! I was able to debug the aws issue, I had mulitple aws accounts I worked in the wrong one and managed to fix it by adding the policy. For google I added "Iam admin" policy into my user and getting another error: ````  Error saving GCP region config
<HttpError 403 when requesting https://cloudresourcemanager.googleapis.com/v1/projects/flash-ward-346914:setIamPolicy?alt=json returned "Policy update access denied.". Details: "Policy update access denied.">
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/cli/cli_impl/init.py", line 251, in load_gcp_config
auth.save_region_config()
File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/utils/imports.py", line 33, in wrapped
return fn(*modules_imported, *args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/compute/gcp/gcp_auth.py", line 32, in save_region_config
service_account_credentials_file = self.service_account_credentials  # force creation of file
File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/compute/gcp/gcp_auth.py", line 68, in service_account_credentials
self._service_account_email = self.create_service_account(self.service_account_name)
File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/compute/gcp/gcp_auth.py", line 179, in create_service_account
return retry_backoff(read_modify_write)  # retry loop needed for concurrent policy modifications
File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/utils/retry.py", line 30, in retry_backoff
raise e
File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/utils/retry.py", line 27, in retry_backoff
return fn()
File "/home/ubuntu/.local/lib/python3.8/site-packages/skyplane/compute/gcp/gcp_auth.py", line 176, in read_modify_write
service.projects().setIamPolicy(resource=self.project_id, body={"policy": policy}).execute()
File "/home/ubuntu/.local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 938, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://cloudresourcemanager.googleapis.com/v1/projects/flash-ward-346914:setIamPolicy?alt=json returned "Policy update access denied.". Details: "Policy update access denied.">

Glad you were able to fix the AWS issue!
Would you be able to double check if the GCP account you're using has iam.serviceaccounts.setiampolicy and
resourcemanager.projects.setIamPolicy? You can do this with the Policy Analyzer in IAM:

Hi Sarah! Skyplane is finally working, I see a speedup of x3 copying 6TB from gs to local s3! Well done!!
Some minor comments:

In the documentation it is written $ skyplane [sync/cp] [local/s3/gcp/azure]://mybucket/big_dataset [local/s3/gcp/azure]://mybucket2/
it should be gs and not gcp (otherwise I see a parsing error).
What fixed it is that i killed the node and restarted it, I also had a double config trying both to give a json key file via GOOGLE_APPLICATION_CREDENTIALS as well as using  gcloud auth login which made a mess. i removed the GOOGLE_APPLICATION_CREDENTIALS and it worked (we have multiple users and it seems the json key policy refused to be for a specific user but was for some system gcloud user).

Glad it's working! Just curious, which permission did you end up adding?
Thanks for pointing these error out with the documentation! I definitely need to add more detailed instructions regarding how to set permissions and how to clear past credentials when switching between users.
Also by local s3, do you mean a s3 bucket mount? We'd actually expect more than a 3X speedup so curious what your setup is.

It may have been an overkill but I have added both:

By local I meant copying from gs bucket into a local s3 compute node ssd drive. I can make some more tests but I got a speedup from 250mb/s to 750mb/s, when working locally inside google cloud i was getting 1Gb/s copying from gs bucket to local ssd drive inside google cloud.

Ok good to know thanks! I'll close issue now that Skyplane's working, but please let us know on Slack or Github if you have more suggestions or feedback!


Cloud Security - detection of role assumable by external account with specified permission

Background
This policy was motivated by https://twitter.com/AlexandreSieira/status/1338477494148718593, pointing out a vulnerability whereby vendors could be granted the lambda:AddPermission via an assumable role. This is apparently a common setup recommended by vendors. The broader issue of external accounts being granted certain powerful permissions is the main point. It would be interesting to review permissions in other services to identify particularly problematic ones, such as permissions that allow an identity to add additional permissions (as here).
Notably, this rule accomplishes the core functionality of AWS IAM Access Analyzer, but in real time, for a single defined service action. The rule could easily be modified to look for sets of related service actions instead of just one. While not nearly as comprehensive as Access Analyzer, the real-time nature of this rule is best reserved for critical service action permissions. In this example case, there is a very small subset of service actions across all AWS services that are analogous to lambda:AddPermission. Many services don't have an analogous action at all.
Changes

This policy detects any role assumable by an account that is not contained in the list of user-configured account IDs that has the lambda:AddPermission permission associated with it, either through an Inline Policy or Managed Policy. While the user could obviously "whitelist" an account they don't control, the intent is that the accounts variable in the policy would only contain accounts controlled by the user.
This policy could become the basis of additional policies that flag different permissions (any IAM permission can be specified with the permission_check variable), or more restrictive triggers that take the allowed resources into account, such as lambda:AddPermission on "Resource": "*" versus just any lambda:AddPermission.

Testing

I think the tests attached evaluate all possible scenarios, to include multiple principals and multiple Managed Policies
This rule is actually running in my Panther dev instance

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

I'm still getting a TypeError: string indices must be integers on some resources, including a test resource that passes in the UI. It's something in the def check_account(resource) that's triggered when lambda:AddPermission is found.

Hey @glerb looks like CI was failing due to formatting
Can you run make fmt?

Hey @glerb looks like CI was failing due to formatting
Can you run make fmt?

So I'm a bit new to the multi-version python world. I have<EMAIL_ADDRESS>installed from homebrew, and have the  latest virtualenv installed from pypi, but getting this error when trying to set up the interpreter (command from make venv):
virtualenv -p python3.7 venv RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.7'
We don't have a lot of Python pros to ask, and/or I wouldn't get an answer in a reasonable amount of time, and I hate to just let this PR sit here for weeks.

@glerb try make install deps fmt from the repo

You can make whatever stylistic/clarity edits you want. I trust your python skills.

You can make whatever stylistic/clarity edits you want. I trust your python skills.

@glerb and I are going to nail this bad boy down on Friday and get this thing through.

Bunch of stuff in this.

My last push somehow destroyed the account lookup functionality. That's fixed.
More efficient looping everywhere.
Better tests
The policy actually has debug-like print() statements in most every block. They print out when you run the tests. For something this complex (and it's only going to get worse when I add in Deny, NotAction, and wildcard perms), I think they're pretty handy for not going insane.
I renamed this to a generic case, because that's really how it should be presented. We might want to make versions of this base policy that look for, e.g., all the things that endgame looks for. Not sure if each permission should have its own rule, or whether we should add the ability to process a whole list of bad permissions. Since policies can't do custom alert titles yet, it might be better to have different policies for each permission.

The too-many-branches linter issue is probably a setting I need cranked up for this one, though it looks like I got a 10/10 anyway.

@glerb any luck getting this over the finish line?


fix hollow button size to match non-hollow button size

https://github.com/zurb/foundation-apps/issues/776

fix has been made in https://github.com/base-apps/angular-base-apps


UI Revamp

The UI looks quite stale in comparison to other websites. It would look better with colors, gradients and animations.

Hi @anubhav1206 👋,
Maybe I could help if you allow me. Note: I don't know how to make animations unfortunately 🙁
Greetings NH

I made a prototype for the new design. Let me know if you want me to change anything. Also, I might need help putting the plan into action.
https://www.dropbox.com/s/tlkqahgcwpvi0f2/goodfirstissues_issue%2352.fig?dl=0
Kind regards NH

Hello @N3v1,
I'm interested in helping with the new design, I'm loving the new prototype. How exactly can I contribute to the project? Should I make more demos or go straight into coding.
Have a good rest of your day!

Hi @t-li-projects 👋,
I'm not entirely sure, but you can jump right into coding. Do not delay in notifying me if you need any help.
Thank you kindly 🙏
Best regards
NH

Hi, if this issue is not fixed yet, I would like to help. Please do assign this issue to me.


TokensAnalyzerTest - More tests on PHP8

@SpacePossum can you take a look at https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/5809? It refactors FunctionsAnalyzerTest heavily.


Map of Byte Array to Charset or CharsetName

I'm trying to decode a MO message using CharsetUtil.decode but I don't know the dataCoding in advance. I can't seem to find a mapping between the dataCoding byte to the Charset or CharsetName
      DeliverSm mo = (DeliverSm) pduRequest;
      byte dataCoding = mo.getDataCoding();
      byte[] shortMessage = mo.getShortMessage();
      // in the following line how can I get the CharsetName from dataCoding
      String messageText = CharsetUtil.decode(shortMessage, "UCS-2");

Am I missing something simple here?

You're confused because SMPP is unfortunately too lenient in my opinion on charsets.  The data coding value is defined in the GSM specs and also implemented here:
https://github.com/twitter/cloudhopper-commons/blob/master/ch-commons-gsm/src/main/java/com/cloudhopper/commons/gsm/DataCoding.java
There are defined charsets which make sense -- but they did leave a spot open called "default" charset.  You actually need to ask the SMSC what charset they use and "config" it on the bind.  We've also found, however, that even if the SMSC sets it to something like GSM, that may not be true.  Most times a carrier doesn't even really know what charset is used and you just need to test it out.  We usually ask the carrier to send an @ symbol, euro symbol, dollar symbol and just deduce the charset they use.  Trust but verify.

One more thing -- we've also found submits and delivers may or may not use the same default charset as well.  I wish the SMPP specs were more strict, but it is what it is.

@jjlauer thanks for your help here.
I just captured some packets from the SMSC we're connecting to and confirmed that they are, in-fact changing the DataCoding based off the message content.
I need to also confirm if they are sending in other charsets as well, e.g. LATIN 1 but I guess that if this is the case I'll need to trust the DataCoding that they send and dynamically decode it using the appropriate Charset.
Would you accept a PR if I put a map in CharSetUtils from the DataCoding byte to the CharSet for dynamic decoding?
I had a look through https://github.com/twitter/cloudhopper-commons/blob/master/ch-commons-gsm/src/main/java/com/cloudhopper/commons/gsm/DataCoding.java and I can see how I can make a new DataCoding instance from the DataCoding byte but it doesn't look like I can get the Charset from there so I can call CharsetUtil.decode with the correct Charset.
See attached files for reference.


Level Editor: Alert if invalid tilecodes are added to the palette

someone typed in lantern which has a sprite so it looked like a valid tile code but then later broke saving because it was in fact invalid. This should alert users early by preventing invalid tile codes from being input.

Fixed


chore: add breaking doc

We're currently processing your upload.  This comment will be updated when the results are available.


Create subscription from current block

Right now by default if I create a subscription, it subscribes from block 0. This is very useful when I need the full history related to something, but if I only care about new events this can be very problematic. I think there is a way to subscribe to a specific block number already, but it would be nice to be able to tell FireFly to just subscribe to everything from this point in time forward.

I think in previous discussions, we've said the default should probably be from the current block, instead of 0. And then we need a way to specifically subscribe from 0 or any other block (via a blockchain-agnostic config, similar to the "location" object for smart contracts).


Question about data sources

Could you provide more info on these data soirees?

Thanks!

Found all the answers here: https://github.com/etsy/411/blob/master/docs/Searches.md


make BlockGroup work on canvas

The BlockGroup should either behave like a regular block in canvas, or be an offscreen buffer for canvas. Either approach should be reliable and work for canvas fallback support.

we can have it behave just like a regular block also - that's easier to do


[BEAM-5653] Allow initialize SdkComponents with existing Components list.

The idea of the fix is to initialize SdkComponents with initial pipeline Components and later extend it.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

R: @kennknowles

R: @swegner @rohdesam

R: @angoenka

R: @pabloem (committer)

Merged as it had been approved by Alex. Thanks.


Connect digital collections via keyword search and for certain examples via subject heading connections

The knowledge panel shows digital collection results that correspond to a keyword search. In addition, on the main search page, if the subject area facet is selected, a subject search (not keyword) is done in digital collections and then the results are returned with images, titles, and links to the entities in the digital collections portal.

For the digital collections subject search, changes had to be made to the views that generate the side bar to enable integration of the subject search results.


Add ability to customize HTTP response error handling

For example, HTTPFile.FileResponder simply return a 404 response on any file errors, regardless of whether its permission or the file is just a directory.
I opened a pull request regarding this with a very in-elegant solution.
It looks like Zewo/RecoveryMiddleware gets us halfway there.

I think RecoveryMiddleware would be fine if we made all of our modules throw errors instead of simply returning a response with a bad status.

yeah! that's the second part of the solution.
provide a complete list of possible HTTP errors. This should be in HTTP I think.
protocol Error: ErrorProtocol {
    var status: S4.Status { get }
}

enum ClientError: Error {
    case badRequest
    case unauthorized
    case paymentRequired
    case forbidden
    ...

    var status: S4.Status {
        switch self {
        case badRequest: return .badRequest
        case unauthorized: return .unauthorized
        ...
       }
    }
}

enum ServerError: Error {
    case internalServerError
    case notImplemented
    ...
}

Then we throw these errors in our modules instead of returning a response, like @Danappelxx said. And as a safety measure. If a user doesn't catch these in a RecoveryMiddleware. And the error gets thrown all the way to the server. We catch errors of type HTTP.Error there and convert them to a regular Response(status: error.status).

Actually, I'd like HTTPServer to not depend on HTTP so maybe a module called HTTPError would be better.

Or we can propose to move this to S4.

This all sounds good to me. As @Danappelxx said, RecoverMiddleware with his route.error{} suggestion sounds great. Will this limit the errors that can be returned by modules like HTTPFile? Does it make sense that it would only return errors that fit into HTTP status codes? Should there be flexibility to give more information? I'm saying this, but I'm having trouble coming up with examples of when that might be necessary.

I think we should strive to only return HTTP errors when we're in that scope. maybe add some more information to HTTP. Error or S4.Error. So we can create a nice looking HTML page for the error. But worst case scenario we catch an anonymous error and convert it to internalServerError.

If you want to make some S4 compatible Middleware or Router, etc. You'd have to throw a common error or else something like Error.badRequest would end up going to a catch all in the receiving application and probably cause a 500.

ok I'll create a PR with this in S4. we can continue discussing what properties S4.Error should have so we can create nicer HTML error pages.

This is possible now. We should inspect where we can throw the HTTP errors and add a default catcher to our HTTP servers.


Soundcloud doesn't scrobble

The icon for the Last.fm scrobbler is displayed at the right end of the URL bar whenever I open and use Soundcloud, however, no song is ever scrobbled from the website. :/ I've tried things like disabling and re-enabling soundcloud scrobbling from the settings, and re-authenticating with Last.fm. However, none of these things worked and still nothing is being scrobbled from Soundcloud. Please help.

The issue is known and will get fixed when #573 is merged.

Sorry for being impatient, but what's the usual timeline on these? It seems like a fairly straight forward change (changing a 16 to 19, and adding fallback). Considering SoundCloud is the 2nd biggest music site and it's been broken for over a month.

Soundcloud scrobbling still broken for me


Adding CORS headers support

Description
Adding CORS headers and configuration option to enable/disable it
Type of change

New feature (non-breaking change which adds functionality)

Testing steps
Regular CI

Isn't it exactly the same code as in aggregator?

@Sergey1011010 yes, it is. Probably when we #42 this code should go away in both aggregator and content-service


Master doesn't bind to the specified host address.

Master always binds to <IP_ADDRESS>, ignoring the host address configured during
installation phase.
See also the following mailing list thread:
http://groups.google.com/group/terrastore-discussions/t/1135c211fc36ceb9

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 2 Nov 2010 at 9:10

Fixed: now specifying the master.host property (or master.host.1 and
master.host.2 for HA installation) correctly binds the master to the specified
host address.

Original comment by<EMAIL_ADDRESS>on 2 Nov 2010 at 10:46

Changed state: Fixed


[AutoRelease] t2-confidentialledger-2022-11-25-65044(can only be merged by SDK owner)

https://github.com/Azure/sdk-release-request/issues/3463
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=2674
BuildTargetingString
azure-mgmt-confidentialledger
Skip.CreateApiReview
true

issue link:https://github.com/Azure/sdk-release-request/issues/3463

/check-enforcer override


In fact, this is an issue, but a comment.

This code is very very helpful for the one just beginning to learn to KVM, thanks for the excellent job.
I just finished testing the MMIO emulation based on your code (changed your code to support KVM_MMIO_EXIT).
And I would like to see your code to have more and more features to show the features/functionalities of KVM.
And here is another example in github for KVM API for your reference.
https://github.com/hitmoon/kvmsample/blob/master/main.c.

Thanks for the comment!  I'm glad to hear you are finding this useful.
And thanks for letting me know about that kvmsample project.  I did search for similar projects, but didn't find anything.  Maybe I didn't look hard enough.  I'll add a link to the README.

I've added a link to kvmsample to the README.  Thanks again!


fix: install lychee to a user owner directory

In a self-hosted context, /usr/local/bin may be not writable by the runner user. Use a user owned folder and add it to GitHub Actions PATH variable instead.
Tests are done. Works as expected.

Thanks @aslafy-z. 👍

@mre any chance this gets released any time soon?
Thanks


Investigate rendering of external content files from PyBaMM repo

https://gohugo.io/functions/encoding/base64encode/

Can be used for CHANGELOG, GOVERNANCE.md, etc.


fix(units): Add a more explicit restart policy

Fixes #4632
Added two stanzas to the unit files for all deis components:
StartLimitInterval and StartLimitBurst. This tells systemd to only allow
a certain number of restarts to occur (StartLimitBurst) within the
specified interval (StartLimitInterval). If StartLimitBurst+1 retry
fails systemd will not try to start the application again. The counter
is reset after the interval.

Added my comment from review in https://github.com/deis/deis/issues/4632#issuecomment-149976182

@jchauncey do you think this could be brought over the finish line in the next week or two so it can go in the Deis v1 LTS release? I know you're busy with Deis v2 tasks, so if this isn't on your near-term roadmap we may want to close it.

I say we close it. This was only useful when we were wanting to monitor v1 in a more meaningful way


Design Look and Feel for the export of Treemap

export the treemap as an image file (e.g. .png or .jpg) which includes the treemap, as well as the JobHopper logo and perhaps a short line of text below it in small font.
The text could say: This visualization shows where {name of occupation} move to when they switch occupation. The visualization was created by Code for Boston’s JobHopper team, using data from 16 million U.S. workers’ resumes compiled by Burning Glass Technologies.

This is also attached to #275, see that issue for additional comments.
Treemap export format.pdf

This is also attached to #275, see that issue for additional comments.
Treemap export format.pdf

closed per @edward-malouf


Transforming Map Keys Fails

I try to transform the keys of a map.
(transform [ALL FIRST] name {:a 1})

should return
{"a" 1}

but it throws a ClassCastException.
On the other hand the following works and transforms the value:
(transform [ALL LAST] inc {:a 1})
;=> {:a 2}

Another approach using a walker works also on keys:
(transform (walker keyword?) name {:a 1})
;=> {"a" 1}

But the walker approach replaces all keywords, not only keys on a specific layer.
Are there better ways to deal with nested maps were I want to transform things on a specific layer? Maybe the approach with ALL, FIRST and LAST is not the best way to do it.

This is already fixed on master and will be released with 0.9.3.

Thanks!

Was this fixed? I can replicate it using specter 0.13.2 (and also 0.9.3). This is the actual code (which keywordizes string keys in a json read string):
(defn load-json [json-str]
  (specter/transform
   [specter/ALL specter/FIRST]
   (fn [k] (keyword (str *ns*) (name k)))
   (json/read-str json-str)))

This works fine for me:
(transform [ALL FIRST]  (fn [k] (keyword (str *ns*) (name k))) {:a 1 :b 2})

What is the class of the object returned by json/read-str?
To look into this I'll need a test case I can reproduce. If you put up a repo with instructions on how to run it I can look into it.

My bad, I am using maven and had the dependency listed in two different places. One with 0.9.2 and the other one with 0.13.2. Oddly enough maven took 0.9.2 and ignored the other version without complaining, but in any case this is not specter's fault.
Sorry for that!


Several issues with Sphinx

On Windows 10, in order to get sphinx running with the given configuration file I had to delete the following line in order to run
enable_star = 1
Not doing so causes the following error on running searchd or indexer
 ERROR: unknown key name enable_star
Also had to do the following in order to get it to run with Sphinx 3.3.1
https://sphinxsearch.com/forum/view.html?id=16497

Download a previous release (3.1.1) and copy the following assemblies from the bin
directories:
libeay32.dll
msvcr120.dll
ssleay32.dll

Can this be added on the windows-setup file?
Lastly there is this problem when archiving gallery, after running TaskRunner and gallery archived, they wont show immediately. and I have to index them manually. This error is displayed when indexer is ran after archiving
DEBUG: [2021-01-24 23:55:13][ExArchiver] Archive complete. Archived 1 galleries.

Sphinx 3.3.1 (commit b72d67bc)
Copyright (c) 2001-2020, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file './sphinx.conf'...
indexing index 'galleries'...
collected 2 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 2 docs, 1.1 Kb
total 0.1 sec, 19.3 Kb/sec, 34 docs/sec
indexing index 'suggested'...
collected 26 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 26 docs, 0.2 Kb
total 0.0 sec, 5.1 Kb/sec, 566 docs/sec
WARNING: could not open pipe (GetLastError()=5)
WARNING: indices NOT rotated.
DEBUG: [2021-01-24 23:55:13][TaskRunner] Finished running task Task_Archive

The last confirmed working version of Sphinx I have for this is currently version 3.0.3-dev, you may want to try this on an older version, as I was never able to get the configuration working as they deprecated enable_star, which was relied upon by the application.

The last confirmed working version of Sphinx I have for this is currently version 3.0.3-dev, you may want to try this on an older version, as I was never able to get the configuration working as they deprecated enable_star, which was relied upon by the application.


feat: windowing operations

Window is an abstraction used in stream processing pipelines to support aggregation operations on a stream of elements. This PR introduces Window abstraction and one of the implementation(s) of that abstraction that allows us to tackle the most common usecase - applying user defined aggregation function over a set of keyed elements delimited by fixed interval boundaries.
we support the following windowing strategies

fixed windows
sliding windows

Also a mechanism to manage life cycle of Keyed windows (fixed interval windows).

The reason why said the sliding implementation is wrong, check the 4min_3min example attached.

So for sliding window, it should be:
t = time.Truncate(interval)
If length >= interval:

From t to t + interval - (length - interval) -> 1 window;
Otherwise -> 2 windows.

Else (length < interval):

From t to t + length -> 1 window;
Otherwise -> 0 window.

So for sliding window, it should be:
t = time.Truncate(interval)
N = int(length/interval)
If length >= interval:

From t to t + interval - (length - interval) -> N window;
Otherwise -> N + 1 windows.

Else (length < interval):

From t to t + length -> 1 window;
Otherwise -> 0 window.

since the sliding window is wrong, let's delete that part of the code and add it later the right way. we are not interested in sliding window as of now.

since the sliding window is wrong, let's delete that part of the code and add it later the right way. we are not interested in sliding window as of now.

removed.


Distribute a prebuilt swift-syntax binary

Description
I think that it'd be great to also distribute swift-syntax as a prebuilt framework with every release, at least for Apple platforms (like https://github.com/realm/SwiftLint does).
We have been successfully experimenting with https://github.com/sjavora/swift-syntax-xcframeworks because the added build time to our CI builds when compiling swift-syntax from the source code is unacceptable.

Tracked in Apple’s issue tracker as rdar://120712265

Strongly agree with this suggestion

Tracked in Apple’s issue tracker as rdar://120712265

@ahoppen can you give an estimation? For huge projects the initial compile time is a huge factor so it can be integrated

@ahoppen would love to know if there's any estimation available regarding a fix, this is a showstopper for our team! 🙏

The challenging part of this is to also support platforms Linux and Windows as well as cross-compilation. We are aware of the issue and will post an update once we have it.

@ahoppen  given that Package.swift supports platform conditionals (i.e. #if os(macOS)), I'd imagine that a temporary patch for Apple platforms (perhaps one pointing to XCFrameworks just for Apple platform builds) would unblock a lot of teams on adopting macros and immediately free up developers from a massive amount of pain experienced on a daily-basis right now.
This issue has been plaguing adopters since the release of Swift Macros. I completely understand and fully support the desire for a fix that supports Linux and Windows, but my understanding is that SwiftPM's current support for XCFrameworks is capable of solving this issue today for Apple platform developers, in a conditional manner that will not disrupt Windows/Linux builds.
There are many folks who've already demonstrated a fix (see InstantSyntax, BinarySwiftSyntax etc.) and the only reason their fixes cannot be widely adopted is because SwiftPM is unable to reconcile a name conflict with swift-syntax (i.e. even if InstantSyntax vends the exact same products as swift-syntax, packages that depend on Apple's swift-syntax will have their dependency on it conflict with InstantSyntax).
Simply adding SwiftSyntax as a dependency can increase Xcode Cloud build times by up to 12 minutes. It has absolute wrecked release build times, and Xcode's inability to reuse build products for even unchanging dependencies across schemes/branches means that this cost is incurred over and over again during a typical work session with a moderately complex project that adopts macros.
Furthermore, Xcode 15's accruing instability necessitates frequently cleaning DerivedData to fix ghost errors/warnings across projects, and Xcode currently does not offer an option to clean the derived data while preserving build products/artifacts for fixed dependencies. This means that swift-syntax and its dependencies is rebuilt in debug mode multiple times as well. This isn't an issue with the swift-syntax project of course, but it is a practical consideration given that a large majority of folks using swift-syntax right now are only using it because it is a practical requirement for Swift macro targets.

Is there any plans to address this for Swift 6?

@ahoppen would love to know if there are any updates on this following all the incredible announcements from WWDC24! 🙏

We would also very much want to have SwiftSyntax provided as an xcframework (since we cannot link directly to GitHub for the sources). We are building it ourselves now with swift-create-xcframework.

Any updates? Waiting for solution for half of a year, struggling with 400% increased build time for medium sized project on day-to-day basis.

The challenging part of this is to also support platforms Linux and Windows as well as cross-compilation. We are aware of the issue and will post an update once we have it.

The most frustrating thing about Swift these days is the ideology that Swift will be one language to rule them all across all platforms when in reality not many outside of the Apple platform community care. This constant unwillingness to support Apple platform only enhancements punishes 99% of the community that uses Swift, which are developers of the Apple platform.

@cameroncooke I echo your sentiment, but I don't think it's productive to speculate on whether this unwillingness or some other reason (which, of course, is unbeknown to us, and that in itself is frustrating). I have many more reasons to believe the opposite, and I have no doubt that they are working on it :)
A few points I'd like to note:

Many, many, many folks have echoed that they encounter this issue daily, and it is nontrivial in how it impacts their build times.
Even after WWDC24, which focused on exciting new developments such as Embedded Swift, Explicitly Build Modules and a myriad of other improvements, there is no word, update or even a hint of what timeline we can expect here.
Even though Swift is OSS, and swift-syntax is OSS, there seems to be no clear way for folks to pitch in here. I know many folks (myself included) who would love to contribute to solving the problem in both discussion and actual PRs, but I don't see any roadmap or checklist to inform a path towards the solution.
This would be a completely different story of-course, if one could truly fork SwiftPM and plug their fork into Xcode, but Xcode uses a private fork of SwiftPM and it is not viable for a majority of Swift developers to switch to a custom fork of SwiftPM interim even if they're motivated enough to temporarily patch the issue.

It's extremely demoralizing to be engaging with this as "an issue of an OSS project". My two cents, if this is an issue which the OSS community cannot meaningfully contribute to and is linked/blocked for whatever (I'm not here to speculate) internal thing, we should perhaps consider making that clear by way of maybe a label and an explicit disclaimer.
I'm only saying this because to the best of my knowledge, the LSG's meeting notes nor the OSS roadmap discussion have touched upon what exactly is going on here (please do correct me if I'm wrong!).

@vmanot

There are many folks who've already demonstrated a fix (see InstantSyntax, BinarySwiftSyntax etc.) and the only reason their fixes cannot be widely adopted is because SwiftPM is unable to reconcile a name conflict with swift-syntax (i.e. even if InstantSyntax vends the exact same products as swift-syntax, packages that depend on Apple's swift-syntax will have their dependency on it conflict with InstantSyntax).

In my opinion, Swift Package Manager should possess the ability to patch the whole dependency graph with a repository fork that has a compatible Package.swift manifest. There are already a lot of package managers (Cargo) in other languages that utilize such overriding possibility for developers who don't want to fork all sub-dependent packages from the dependency graph. For security and readability reasons, it would be necessary to introduce an explicit patch delimiter, i.e., Package.Dependency.patch(url:from:). As such, a top-level, most dependent package will have the ability to alter any of its dependencies. Is anyone in contact with SPM devs?

SwiftPM should support this already using dependency mirrors: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0219-package-manager-dependency-mirroring.md

SwiftPM should support this already using dependency mirrors: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0219-package-manager-dependency-mirroring.md

What a surprise! I've found out that solution at the same time you replied. ^^
It seems that the problem is actually solved for everyone willing to opt compilation terror out. I would love to switch to prebuilt swift-syntax on debug builds while opting official package in prerelease and release builds.

compilation failed at: switch covers known cases, but 'AccessorBlockSyntax.Accessors' may have additional unknown values

You might also have luck if setting BUILD_LIBRARY_FOR_DISTRIBUTION to NO in your project's build settings, although it might cause other kinds of issues.

@lyzkov to reply to your first comment:

In my opinion, Swift Package Manager should possess the ability to patch the whole dependency graph with a repository fork that has a compatible Package.swift manifest.

I agree.
I also think that what you're talking about and what this issue ("Distribute a prebuilt swift-syntax binary") is about are orthogonal to each other.
The issue here is that there exists (unless I'm missing something, please correct me if I'm wrong) the means to distribute swift-syntax in a binary format for Darwin platforms.
The last reply we have (unless I've overlooked a message in this thread) on the matter of the issue of why Apple isn't vending XCFrameworks today is from @ahoppen:

The challenging part of this is to also support platforms Linux and Windows as well as cross-compilation. We are aware of the issue and will post an update once we have it.

That message was sent 118 days ago, and it mentions that we'll receive an update once they have it, so my understanding today is that there is no update on the matter.
A fairly well-engaged-with post, "Compilation extremely slow since macros adoption" was opened 309 days ago where the last relevant reply was from John McCall (here) stating:

The LSG is aware that there are build-system issues causing a lot of pain for macro adopters and their downstreams. From an abstract language perspective, I don’t think any of those problems are particularly challenging to the point of requiring an overall design change to macros; the project just needs to put in the work to fix them. I can’t tell you when that will happen, though, which I know is not a satisfying answer.

This is, to the best of my knowledge, the current state of the issue.
Clients of swift-syntax, which is practically a mandatory requirement for leveraging macros in Swift (this is still the case as of Swift 6, unless I've missed something) will be required to build all of SwiftSyntax for each clean build.
I just ran cloc out of curiosity on the Sources folder from the tip of main as of writing this comment.
vatsal@Vatsals-MacBook-Pro Sources % cloc .
     347 text files.
     343 unique files.
       7 files ignored.

github.com/AlDanial/cloc v 2.02  T=0.90 s (381.4 files/s, 192826.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Swift                          287          17546          23479         129239
Markdown                        21            440              1           1595
CMake                           19             43            116            385
XML                              6              0              0            300
C/C++ Header                     7             30             96             68
C                                3             11             22             35
-------------------------------------------------------------------------------
SUM:                           343          18070          23714         131622
-------------------------------------------------------------------------------

Sources checks out at 129239 (~130K) lines of Swift code.
The effective lines of code being compiled will be far lower, but even if we halve it, that's ~65K lines of code.
Compiling ~65K lines of code for a source dependency in order to leverage a fundamental language feature that was released more than a year ago is the norm, at the time of writing this.

@ahoppen can we please get some more insight on this? You mentioned that it's being tracked internally via a radar but that's not very helpful to outsiders (I believe these things that directly affect developers should be discussed publicly, that's kind of the point of swiftlang now being a separage GitHub org, isn't it?). I'd love more transparency on this, at least knowing what's the blocker right now since the community seems willing to contribute or solve issues.

@lyzkov you're facing that error because when building from source, the compiler is aware of swift-syntax's enum cases. When compiled, I'm assuming these enums are not annotated with @frozen, so the compiler does not know whether there might be other cases in the future that are unhandled.
IMHO this should be a warning and not an error (I'm surprised that it's an error).
You should be able to work around it by using the unsafe flag -disable-nonfrozen-enum-exhaustivity-diagnostics (see here).
Feel free to reach out to me on Twitter at @vatsal_manot, I respect and am happy to support folks trying to solve this problem today.

I've tried to pass -disable-nonfrozen-enum-exhaustivity-diagnostics in Other Swift Flags, but it seems to work only for top-level package/project scope.
Then I've added it to forked binary, unfortunately, because unsafe flags are rejected by SPM for published packages. They are only allowed in dependencies under local development.

Every so often I hear people from the fruit company muse why everyone thinks that Swift is a language controlled by Apple and that the only reason it gets used is for iOS development. After all, there's a WWDC session about putting Swift on a system that doesn't have a MMU, the GitHub org is hosted at "github.com/swiftlang" now, there is now more than one non-Apple person on the Core Team, and The Browser Company did a tour de force of shipping it on Windows. The community is healthy and vibrant, the haters were mistaken, and Swift on the Server is a real thing not being propped up by IBM Amazon somehow Apple's security team?
Then, of course, someone files an issue against one of the cross-platform bits and a Radar link shows up. I mean, I get it, if it's not in Radar it doesn't exist, there's an EPM who really wants to mark things as P2 – NMOS, I know the drill. If you want to have your own funny number in your internal bug tracker that is completely fine. All of us have our own little rituals too and some of us get paid to partake in them :)
However, the converse is also true: for open source projects, if it's not done in public, then it doesn't exist. If you want to figure out when this will ship in Xcode you're welcome to do that in Radar. (Well, I would really like that you didn't. But that's your own proprietary thing, and you have WWDC slides to make, that ship has sailed.) But for something like this the discussion should really happen in the open. When the concern is "we would like to figure out how to ship this on Windows and Linux", first of all, "we" is not "people at Apple who have access to Radar" but "the people who contribute to Swift" and the "figuring out" should involve the people who actually use Swift on Windows and Linux. I assume this is a hard problem™ and Apple has priorities™ but one of the nice things about an open source project is that other people might both care more than you and have interesting new ideas on how to fix things. If you say you are working on it, then everyone is going to wait for you to do it, lest they get co-opted by your solution. But if that is happening internally we have no insight into what is going on there, so now not only is nobody going to want to put effort into fixing this but also they are going to bug you continuously for updates because you said you were the ones doing it. If you don't actually have the ability to share those, nor are you able to solicit feedback on what you want to do, well, now the project isn't very open anymore.
I don't want to pile on you specifically, @ahoppen. This is far from the only bug where this kind of thing happens, and this is probably just the internal policy dictating what people who aren't Ted are supposed to say in a public issue tracker. But I do hope that someone goes back and critically evaluates why the discussion for "how can we make this work for Windows" is happening in a private, Apple-internal bug tracker. If you want to present Swift as a community project then it really ought to be tracked here and then you can have your bots mirror it back into Radar if you want.
(Yes, I understand that this is a Wendy's; I am just too lazy to actually post this where it probably belongs. My apologies for taking this issue on a tangent.)

@saagarjha The topic of discussion is binary, precompiled library distribution that does not violate open-source licenses. That's what everyone is asking for. It should not contain complaints about Apple's policies. We should focus on finding a secure way to provide package mirrors that ship binary versions compiled on trusted machines and ensigned by a trusted developer. Till now InstantSyntax is closest to a satisfactory solution for Apple Developers as it provides a shell script for generating precompiled XCFrameworks. You should not be irritated about not all platforms included, i.e. WASI. It can be achieved by other mirrors.
In my opinion, the main worry is how to ensure that the binary version of the package is officially sourced in swift-syntax. Maybe every developer should precompile the library target on their own machine for the first time library is built? It can be achieved with the Package Prebuild Tool Plugin. Then 130k lines of code will not be indexed by Xcode.

@lyzkov I think his comment (echoed by others such as @PatrikTheDev) speak to confusion as to why this issue is being tracked internally by Apple as opposed to being openly discussed here.
The topic of discussion is the issue of swift-syntax not being distributed with prebuilt binaries.
We don't have a clear answer on this yet. We don't have a roadmap, we don't have a sense of what is involved in making this decision.
Is this an issue of the OSS project that is swift-syntax or not is a natural question and very relevant question.

@lyzkov I think his comment (echoed by others such as @PatrikTheDev) speaks to confusion as to why this issue is being tracked internally by Apple as opposed to being openly discussed here.

I think that Apple is interested in Swift's macro adoption by Xcode. Right now I feel like it's supported but not integrated well. Maybe that's the cause of the issue being tracked internally?

I think that Apple is interested in Swift's macro adoption by Xcode. Right now I feel like it's supported but not integrated well.

The problem is that the integration in Xcode only really works with macros provided by Apple in their SDKs. As soon as you try to use a third party or even your own macro you're left with the huge burden of needing to rebuild swift-syntax every so often. And this is the reason many of us developers are upset. Apple praised macros to be the next big thing but they practically aren't usable right now.

@lyzkov swift-syntax is part of the Swift OSS effort. Binary distribution is an issue that affects even those folks who have never used Xcode.

@vmanot Is swift-syntax capable of being a building block for swift-analyzer that could replace Xcode Indexer the same way swift build is intended to replace xcodebuild? Maybe that’s the idea for deep integration?

The most frustrating thing about Swift these days is the ideology that Swift will be one language to rule them all across all platforms when in reality not many outside of the Apple platform community care. This constant unwillingness to support Apple platform-only enhancements punishes 99% of the community that uses Swift, which are developers on Apple platforms.

This sentiment saddens me a bit. Sure, it's a bit of a chicken-and-egg problem to make Swift a viable cross-platform language, but the teams are doing a lot of great work on making things work across other platforms and just "piling another hack" to make things work on Apple platforms feels like the opposite of that direction.
I agree that the situation is not ideal, but I don't think that taking the time to get this right is necessarily the worst option.

I don't think that taking the time to get this right is necessarily the worst option.

@fwcd no one is arguing that taking the time to get this right is a bad option, but the discussion of whether that is mutually exclusive with platform conditional solutions has not taken place.
I personally disagree with your characterization of it as "piling another hack" and am happy to elaborate on why, but that is not the point. I'd love to engage in an actual concrete, thoughtful discussion around this as opposed to recycling the same, milquetoast comments (it's a hack, it's not a hack, it's fine, it's not fine etc.).
The status of "we're fixing this, the discussion around that is internal, check again later, we don't know when!" is the problem. We have not been given a concrete explanation, and we might not be given one for quite some time. Many of us here are just seeking clarity on whether this is an issue within the purview of the OSS Swift effort or an internal Apple thing, and the lack of clarity there is frustrating and unproductive for all involved.

(Disclaimer: I'm only tangentially involved in any of this and not at Apple, so this is mostly speculation)
My impression has been that there is a strong interest in integrating swift-syntax into the toolchain itself, similar to what was done with the Regex DSL. But I'm too unfamiliar with the precise logistics of how the Swift toolchains are packaged to really say for sure whether that would be an option.
From what I understand the reason for depending on swift-syntax as a package for now is that it affords the compiler people a bit more flexibility with respect to the API by letting clients depend on a swift-syntax version independent of the compiler version. This thread might be worth reading: https://forums.swift.org/t/macro-adoption-concerns-around-swiftsyntax

@fwcd all of that is congruent with the idea of taking advantage of SwiftPM's existing support for leveraging XCFrameworks on Darwin platforms (conditionally) right now.
The historical context of why swift-syntax is separate and not integrated with the toolchain is important, yes. IMO the more relevant discussion is around solving a practical issue with means already demonstrated by the community. The premise of it being independent from the compiler is already accepted by those engaging in these discussions, I'm not sure how that's directly relevant, but maybe I'm missing something.

As a follow up on @DougGregor's statement of the desire to have a cross platform solution and have a solution that benefits the entire ecosystem, I'm taking a look at this from the SwiftPM side and am investigating various ways we can distribute prebuilt binaries. There are a few options, all of which have issues that need to resolved, which is why this is taking time. I'm digging into one avenue to see if it's feasible which might solve this more generally in the long run with a preliminary solution that should help SwiftSyntax directly. Once I'm sure it could work, I'll present the list of options in a forum post and we can continue the discussion there to get more eyes on it.

I have been putting the finishing touches on a proposal the last couple of days now that the Server-Side Swift conference is over. I talked to lots of people there who shared the same concerns. I should have something to share shortly.

@dschaefer2 If you're planning on sharing the proposal in a different forum, could you link to it here when it's ready?

Thank you for your work @dschaefer2! Eagerly awaiting an update.

Me too 😄. I am working hard on a design right now. In the end, the strategy remains the same as discussed here, produce pre-built binaries of swift-syntax for all our supported host platforms and store them somewhere. SwiftPM would then detect when these are available to meet a package's dependency requirements, download the binaries securely, and use them in the build graph. I'd also like to cache them locally like we do with the source so it only needs to fetch new ones.
It's definitely my focus now that I have a handle on some of the bigger Windows SwiftPM issues. I'll provide a more detailed request for comment once I have a good idea of the code changes needed in SwiftPM to support this. At a high level, it feels like a good path though.

@dschaefer2 I'm just wondering if changes in SwiftPM will enable us to distribute pre-built binary for other loc-heavy libraries? It would be very helpful for system frameworks that are still under construction but whose basic functionality is mature enough to not update the package with each version increment.

I'm just wondering if changes in SwiftPM will enable us to distribute pre-built binary for other loc-heavy libraries?

I certainly have an eye on that. At the very least we'll learn from this but I am hoping it's a path towards making this a more general solution.

Any update here? :)

Working on implementation details. Essentially what I said above but focused on macros only, not the general prebuilt binaries use cases to keep things simple for this first cut.

Hi @dschaefer2!
How would this support various different Linux distributions, statically linked? Build once for the current machine and cache on the machine per user?
Just thinking out loud; there is https://github.com/swiftlang/swift-package-manager/issues/7035#issuecomment-2321960117, and a branch from @xtyxtyx which adds SPM POC support for binary artifact bundles. The sample demonstrates distribution of binary artifacts for macOS, Linux, and Windows.
Would this help solve the swift-syntax compile time issues?

As I mentioned, this initial solution is only for macros which is simpler since I only need to worry about building for host platforms, i.e. the same platform the toolchain is running on.
There is clear desire for expanding this more generally and we'll have to deal with those additional complexities when we get there.

Any Update？It has been nearly a year since this issue was raised. Although it is a complex issue, it is still recommended to solve it gradually, giving priority to the binarization problem of Apple platform, and postponing the consideration of other platforms?
Too long discussion seriously affects the use of Swift Macro in the project, which actually becomes a chicken rib


Added composite identifiers to get field of struct.

Closes #603
Note that this only works with full identifiers, i.e. SELECT test.c1.field FROM test.
The DataFrame API is named Expr::get_field, just to make it explicit. We could add col("c1")["field"] or something.

This does not work for scalar values yet,  #602 tracks that.

@jorgecarleitao  shall we merge this? Do you need some help wrapping up the additional tests

@jorgecarleitao i am available to help finishing up whatever is blocking this PR from getting merged.

Closing stale PRs to keep the PR list manageable. Please reopen if this is a mistake (it seems like maybe someone else could pick up this work and drive it forward when needed).
I may do so if/when we find calculating some of our timeseries analytic functions in IOx are taking a long time, but we haven't seen that yet


Some one can help me to identify what is wrong. (changing font size of an element)

Challenge Change the Font Size of an Element has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.

I've just encountered the same problem. Can someone help?

@musekwa @jacobsteenhuysen  post your code somewhere.  In the original example above it might be something as simple as that last "" is capitalized but in the stylesheet is references a lowercase 'p'.  If you change that lowercase so everything matches, then that should fix it.  i passed the test.

Hello dear robbiegod, here is the code:

I've tried to change the  from lowercase to upper one, but it couldn't pass

Finally, it worked. I was mispelling the "font-size" to "fonte-size"...

@musekwa @jacobsteenhuysen @robbiegod I hope you all kept coding, and now look back and laugh at this 😀 and also be proud that you did not quit!


Add support for tracing

Draft PR to compliment soto-project/soto-core#443
This fork will evolve over time as we add support for more services and work out a better way to do so without manually hacking everything in.

Closing this. We still need to find a solution for DynamoDB and S3 tracing attributes though


⚠ Bofrost.de has degraded performance

In 5726237, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1390 ms

Resolved: Bofrost.de performance has improved in 8cf00f2 after 6 minutes.


Changed facility source to ODS from HISImplementation

Changed facility source for AggregateFACT_CT_DHIS2

@cbrianbet I think the query will still do without collation.

I got this error when I tried


Modify mbed ignore

Do not ignore 'app' directory since it provides the interface for constructing X509 certificates and more.

The content of the 'app' directory being used by the provisioning team

@Patater FYI


URL might not be parsed

Connection with issue(s)
Resolve issue #983
URLs like Android intentions might not be parse. So you need to pass URLs as string.

To Do

[x] double check the original issue to confirm it is fully satisfied
[ ] add testing notes and screenshots in PR description to help guide reviewers
[ ] request the "UX" team perform a design review (if/when applicable)

To manage URLs with custom schemas, you need to implement your specific logic by yourself, for example, implementing the shouldOverrideUrlLoading event.
For example, using it with the url_launcher plugin:
shouldOverrideUrlLoading: (controller, navigationAction) async {
  var uri = navigationAction.request.url!;

  if (![
    "http",
    "https",
    "file",
    "chrome",
    "data",
    "javascript",
    "about"
  ].contains(uri.scheme)) {
    if (await canLaunch(url)) {
      // Launch the App
      await launch(
        url,
      );
      // and cancel the request
      return NavigationActionPolicy.CANCEL;
    }
  }

  return NavigationActionPolicy.ALLOW;
}


Adding support for Azure AutoML Mlflow Object Detection Models

This PR adds a wrapper for object detection models trained using azure AutoML. These models are wrapped using an MLflow wrapper and this PR adds code to process outputs from this mlflow model. The class list and size of images were additional inputs needed to ensure the outputs were standardized across the other object detection wrappers in the repo.

Another commit added these changes. Closing this.


Multiple ollama_llama_server process are created

What is the issue?
In the process of use, after a period of time through nvidia-smi view, there will be multiple processes using the GPU, but only one of these processes actually works. You can confirm this by using the ollama ps command

Refer to the above pic, only the 2115700 process is valid, and it is clear that there are two other processes 1990922,2036868 that occupy a fixed size GPU memory, and another process 2117261 is still running..
OS
Linux
GPU
Nvidia
CPU
Other
Ollama version
0.3.5

Refer to another set of data
(base) xxng$ ollama ps
NAME            ID              SIZE    PROCESSOR       UNTIL
codeqwen:chat   df352abf55b1    6.8 GB  100% GPU        27 minutes from now
qwen2.5:3b      357c53fb659c    3.1 GB  100% GPU        2 minutes from now
(base) xxng$ nvidia-smi
Wed Dec  4 15:13:24 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-PCIE-40GB          Off | 00000000:17:00.0 Off |                    0 |
| N/A   31C    P0              58W / 250W |  38781MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1573      G   /usr/lib/xorg/Xorg                            4MiB |
|    0   N/A  N/A    708055      C   text-embeddings-router                     1586MiB |
|    0   N/A  N/A   1330781      C   /app/api/.venv/bin/python                  2812MiB |
|    0   N/A  N/A   1398516      C   Model: whisper-large-v3-turbo-0            2292MiB |
|    0   N/A  N/A   1630783      C   python                                     7158MiB |
|    0   N/A  N/A   1990922      C   ...unners/cuda_v11/ollama_llama_server     6692MiB |
|    0   N/A  N/A   2036868      C   ...unners/cuda_v11/ollama_llama_server     6692MiB |
|    0   N/A  N/A   2117261      C   ...unners/cuda_v11/ollama_llama_server     6692MiB |
|    0   N/A  N/A   2130892      C   ...unners/cuda_v11/ollama_llama_server     3226MiB |
|    0   N/A  N/A   3573426      C   text-embeddings-router                     1554MiB |
+---------------------------------------------------------------------------------------+
(base) xxng$ ^C
(base) xxng$ top -b | grep ollama
1950229 ollama    20   0 9378976<PHONE_NUMBER>00 S  23.5   0.5   4:21.30 ollama
1330180 ollama    20   0   33580   4652   2028 S   0.0   0.0   0:44.99 redis-server
1411787 ollama    20   0 2910268   4440    500 S   0.0   0.0 871:07.99 mysqld
1990922 ollama    20   0   46.2g   1.3g 396956 S   0.0   1.0   0:03.92 ollama_llama_se
2036868 ollama    20   0   46.1g   1.3g 397020 S   0.0   1.0   0:03.44 ollama_llama_se
2117261 ollama    20   0   48.2g   1.3g 397696 S   0.0   1.0   0:03.19 ollama_llama_se
2130892 ollama    20   0   43.0g   1.6g 627340 S   0.0   1.2   0:02.66 ollama_llama_se

Below is a two-hour log attachment : Please remove the suffix .txt because the original log is about 150M
ollama_new.7z.txt

i suppose it could be simple subprocesses .. i think they can share much , also in metadata

I have a similar issue, windows?. If for whatever reason, ollama silently crashes and restarts itself no errors or, restarts itself and the log resets also, so ollama also skipped the errors debug logs etc? but it leaves orphaned ollama_llama_server processes (eg: using llava with context / message history from llama.3.1 / 3.2 or using continue vscode and something goes wrong).  I fixed my issue by tracking the pid and removing orphaned server processes. slog.Debug("detected orphaned llama server processes", "pids", orphanedPIDs)

re-create the issue: Using the generate endpoint with context[] (yes, I know the context is being depreciated) does it easy! first use llama3.1 and using the context / message history with llava, but there are many ways this issue is caused. Good luck.  heres my attempt lol. https://github.com/YonTracks/ollama-yontracks/commit/9eb047d01c3d45acb6200f9c6dfe1358e0ae2047

i suppose it could be simple subprocesses .. i think they can share much , also in metadata

Thanks, but it seems to be more than that, looking at the log seems to keep trying to load the same model repeatedly.. Ollama output a lot of logs in a short time, but I don't understand them very well.

re-create the issue: Using the generate endpoint with context[] (yes, I know the context is being depreciated) does it easy! first use llama3.1 and using the context / message history with llava, but there are many ways this issue is caused. Good luck. heres my attempt lol. YonTracks@9eb047d

OK. thanks. glad you already have a solution. Simply analyzing the logs, ollama rocesses are being rebuilt all the time, and then orphans are left behind. I will also try your suggestion

did u have off on's on the internet ?
i noticed recently , in phone termux , if i let model download and then switch away from termux and off inet , and on it again ,
.. go termux , the dl restarts at 0% or saved_last_state , just as the cmd started ...
maybe its somehow related ..

windows: vscode: cotinue:
This happens with windows continue and vscode and also loops at times. I think mostly a continue and vscode causing issue. I have no issues or surprises using my own UI unless I purposeful error causing actions lol. I will keep testing and learning what is happening. trying to get the server.logs at the correct time of the issue is hard lol.
ollama-embedding-loop.txt
with 0.5.3 I can capture the logs lol. seems to be trying to embed the whole repo lol.
I can fix the orphaned server processes but still looping issues related I reckon so only temporary for now (I regularly test official ollama repo via OllamaSetup.exe). I am trying to trace the initial cause and reason's, the other loop I found was expired runner retry and looping same until timeout or crash.

Found how to re-create the current embed looping issue:
So far with vscode, windows 11, nomic-embed-text and continue 0.5.3 but similar will earlier releases.
ollama installed via OllamaSetup.exe.

First (with ollama not running) open vscode (current test is ollama fork repo, but I have had same / similar with other projects), then run ollama via the windows ollama icon (will see the tray icon appear).
Open the server.log via tray icon view logs and open with vscode.
no issue yet.
inside vscode currently(ollama repo), right click dist/OllamaSetup.exe Reveal in File Explorer
looping... gpu max nomic model vram 1.9gb or something but runs 90+% and cpu starts to quickly climb.
continue-config.txt

so far this is a easy way to have this issue happen, but is has happened to me without using vscode(but loaded the logs into vs code to view logs only, so not always large repo embeddings), and also while using a local dev build and serve or start.
I don't think caused by ollama and if everything is correct and ollama is used correct I think all is good, but if something is not then issue / similar issue.
For now for me I just keep using my fork with orphaned server fix, and monitor the resources if leaving ollama running or just let it crash lol when it restarts, the orphaned server fix catches the orphans, if looping my fans kick in lol. Good luck
If I find anything worthy, I will share. cheers


Nodes metadata missing

There seems to be another issue. Could you please check why Retrieved Nodes and Edges are undefined? It doesn't matter the scenario (cluster or otherwise), the information seems to be missing. The data is obtained from NetworkStatistics lambda function. The code is placed under docker/networkstats. I think the info is not parsed at all, so we are seeing it as empty.
Possible reasons to look into

datafetcher amplify function
DataBar/Header.tsx component --> Selected itemId is not having the information that we need
https://github.com/maastrichtlawtech/case-explorer-ui/blob/prod/components/CaseLawExplorer/components/DataBar/Header.tsx

What it should look like:

@davidwickerhf - https://github.com/maastrichtlawtech/perfect-graph/blob/f3f4fd2555f0329650b8bc7e2b54011f8e0485c3/src/components/GraphEditor/DataBar/index.tsx#L131 The "Node Information" is taken from Perfect graph. From the commit history (git blame), this was there before - { id: ${item?.id}} which is now changed to 'node information'. Maybe the root cause?

Is this related?

Error is in Lambda function. The code where it fails - https://github.com/maastrichtlawtech/case-explorer-ui/blob/e218279491d1a3dfcc5fe91bcc6d7537d5678009/amplify/backend/function/datafetcher/src/index.py#L8
After checking a couple of things, there was an env variable missing in the lambda function. The environment variable was called DDB_TABLE_NAME. Updated the lambda function with the appropriate env variable.
Validated it on the platform. Issue resolved.


Web content filter for 3rd party browsers isn't working after upgraded to 4.18.23050.3.

Web content filter for 3rd party browsers isn't working after upgraded to 4.18.23050.3.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 61d65c37-bbb6-5c7a-8405-5994badd0b23
Version Independent ID: 61d65c37-bbb6-5c7a-8405-5994badd0b23
Content: Microsoft Defender Antivirus security intelligence and product updates
Content Source: microsoft-365/security/defender-endpoint/microsoft-defender-antivirus-updates.md
Service: microsoft-365-security
Sub-service: mde
GitHub Login: @denisebmsft
Microsoft Alias: deniseb

@sparkf1 Thank you for your feedback. Sorry to hear that you have a technical issue. I'd recommend you contacting Support via the need help section on the admin center, so that a support engineer can fix or raise a bug with the Product group to fix this.
Hope this helps!
Thanks
Sri


Use go 1.14.x as the default go version

This has been resolved by https://github.com/paketo-buildpacks/go-compiler/pull/57. That PR includes fixes to the integration suite that were broken by this change.


Add Java Bindings

I thought this was just the boilerplate that people could add to the files they contribute. The year could be anything, that's why I didn't change it.

Would love to get input from @reinecke and others who have been tracking the java PR on the main repo to weigh in on anything that might need to be added to this repo.  Thanks!


Add missing API methods

API reference

Environment variables names has been changed, please update it in pipeline

Environment variables has been changed, please update it in pipeline

the credentials are parsed from the installation logs on the CI here

Environment variables has been changed, please update it in pipeline

the credentials are parsed from the installation logs on the CI here
Can you edit this, because I'm not strong in using "sed"?

It's already deployed to pypi :smiley:
Thanks for contributing :rocket:


HttpServer trait?

I found http client in embedded-svc it's a Connection wrapper
https://github.com/esp-rs/embedded-svc/blob/ad0016b5e1d74d447c929a7cb78dc3ba358fc676/src/http/client.rs#L7-L11
But is there any http server trait?
reference to esp-idf-svc:
https://github.com/esp-rs/esp-idf-svc/blob/2191e60cb8b85fba5e64bd09a55028955d38792d/src/http/server.rs#L405-L412
https://github.com/esp-rs/esp-idf-svc/blob/2191e60cb8b85fba5e64bd09a55028955d38792d/src/http/server.rs#L438-L440
Maybe we can use trait code as follows?

pub trait Server<'a> {
    type Conn<'r>: Connection;
    type HttpServerError: std::error::Error;

    fn handler<H>(
        &mut self,
        uri: &str,
        method: Method,
        handler: H,
    ) -> Result<&mut Self, Self::HttpServerError>
    where
        H: for<'r> Handler<Self::Conn<'r>> + Send + 'a;

    fn fn_handler<F>(
        &mut self,
        uri: &str,
        method: Method,
        f: F,
    ) -> Result<&mut Self, Self::HttpServerError>
    where
        F: for<'r> Fn(Request<&mut Self::Conn<'r>>) -> HandlerResult + Send + 'a,
    {
        self.handler(uri, method, FnHandler::new(f))
    }
}

Such a trait used to be there long time ago.
I have removed it, because this trait is impossible to implement without using alloc. And you need alloc to turn the generic H: Handler handler into a dyn Handler instance, by potentially using boxing, as in Box<dyn Handler>.
Since embedded-svc strives to have all its traits implementable in no_std and without heap allocations, I therefore removed it in later versions.
While this trait might be useful, its usage is very limited - as in it can only abstract the code that registers your handlers. Abstracting the handlers themselves is way more useful, as that way you can re-use them across many HTTP servers.


cmake: add FindCLI11.cmake

Add CLI11 finder in cmake/modules
Refactor external/CMakeLists.txt to only fetch CLI11 when it can't
find it.

Coverage increased (+0.03%) to 68.206% when pulling cf4d7e6a00895eb5127c31f575e6fecdd93303ed on rrojo/refactor-cli11-use into c447e1442b718e7fa26668a37ab2ca18796f1e30 on master.


Module reorganizing

With library users in mind, we need to reorganize our module names

placing all modules inside a common root, like Test.Plutip.XYZ
separating public and internal modules

Resolved with #21