Harbor Graduation Review

SIG-Storage Graduation review request of Harbor as per cncf/toc#311

/assign

Harbor is a DIY environment-agnostic container registry. Goal of my review is to evaluate the Harbor project for graduation purely from the storage perspective. Any non-storage concerns should be be in https://github.com/cncf/toc/pull/311
From the storage perspective, I want to see a tight integration with Kubernetes and the ability to deploy in an HA way on an arbitrary cluster (without tight coupling with underlying storage).
Based on https://github.com/goharbor/harbor/blob/master/docs/1.10/install-config/harbor-ha-helm.md#usage Harbor has the following storage dependencies:

Application Data

StorageClasses/PVCs OR Object Storage

Images & Charts Data

PostgreSQL database
Redis cluster

My observations:

Uses of StorageClasses/PVCs for Application Data is good. It ensures the deployment of Harbor is independent of any specific storage implementation.
Optionally use of an object store instead of PVCs is a nice option for users. While it does tie the user to a specific implementation, based on https://github.com/goharbor/harbor-helm#configure-the-way-how-to-persistent-data the following object stores are supported: azure, gcs, s3, swift and oss, and object storage is not required and is only provided to make it easier for users who can't provide multi-writer file, which seems reasonable.
The dependence of Harbor on PostgreSQL database and a Redis cluster could be problematic. I know the TOC has had some concerns about the neutrality of dependencies (while I don't have an issue with that, it may come up in the main review). I am concerned that based on this, the deployment and set up of these required dependencies is left as an exercise for the user. @michmike can you verify if that is the case?

@saad-ali thank you for the review. To answer your question, the Harbor Helm Chart (or our docker-compose deployment method) will deploy single instance of redis and postgres automatically. To deploy a HA redis/postgres, users need to set up them externally. We are currently working on a Harbor operator that will help setup automatically HA redis and HA postgres.

we are also working in the near future to possibly replace postgresql with CockroachDB

I would predict that the TOC would look negatively on a requirement on a non-open source project like CockroachDB. Having a pluggable DB layer with multiple options (some open source, and some proprietary) could make sense. But actually replacing PostgreSQL (a well-regraded open source database) with CockroachDB (a source available database) would be problematic.

Having a pluggable DB layer with multiple options (some open source, and some proprietary) could make sense

hi Dan, that's why i mentioned "possibly" in my comment. we would first have to do our due diligence, and this has not happened yet. of course we value the opinion of the TOC in this matter.

Thank you to the Harbor team for providing more information.
CNCF SIG Storage Due Diligence Report
Observations
The due diligence observations are summarized above in https://github.com/cncf/sig-storage/pull/52#issuecomment-587260290
Recommendation
After completing due diligence, members of CNCF SIG Storage have raised some concerns that are outlined below. Some members of the SIG believe that we should hold off on graduation until they are addressed, others don't see them as blocking concerns. We leave the ultimate decision up to the CNCF TOC.
Areas of Concern
Lack of Out-of-Box HA
Deployment of the dependencies, Redis and Postgres, in high availability configuration is left up to the user (not supported by the Harbor Helm chart).
Counter Argument
The harbor team had two counter arguments

Postgres and Redis have their own Helm charts that support HA deployments.

The argument is that for customers that desire this, they either already have their own HA deployment or can use the  Postgres and Redis Helm charts to achieve HA.

Application-Level Async Replication

Two harbor instances can be set up to asynchronously replicate a registry.
This makes a harbor instance more tolerant to whole cluster/site failure.
Since replication is async (not sync), there is still a chance for data loss, but this may be sufficient for some (based on desired SLO).

Dependence on Postgres and Redis
In the past, members of the CNCF TOC have expressed some concerns about the neutrality of dependencies, in general extensibility is desired.
Counter Argument

Given that both dependencies are open source, it's unclear what the concerns are.

/close

Can SIG-Storage update the consolidated SIG review template with its assessment http://bit.ly/harbor-graduation-dd?  Thx!


Delete A record

Hi,
Is there a way to delete an existing A record.
I could not find one, also tried setting ttl = 0 in client.put_a_records() but
that seem to have no effect.

Possibly this API call with a blank A record...
https://developer.godaddy.com/doc#!/_v1_domains/recordReplace
I haven't tested this myself, but you could try to use Client.put_a_records('some.domain', []) and see what happens.

Hi eXamadeus,
My requirement is that I just want to delete a particular record.
Passing an empty list means i want to delete all records for given domain.

Ahh sorry, I misread your comment.  I guess you'd need to get them all and "put" back that list minus the one you want to delete.  You should be able to do that with the Client.
Sent from my iPhone

On May 31, 2016, at 6:24 PM, Sidhesh-telsiz<EMAIL_ADDRESS>wrote:
Hi eXamadeus,
My requirement is that I just want to delete a particular record.
Passing an empty list means i want to delete all records for given domain.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi eXamadeus,
Initially I only have @ A records. Then i add another A record and I see @ and new A record.
If the behavior which you mentioned was the case then my root itself would get overwritten.
In [16]: client.get_a_records(domain='icloud-net.com')
Out[16]:
[{u'data': u'<IP_ADDRESS>', u'name': u'@', u'ttl': 600, u'type': u'A'},
{u'data': u'<IP_ADDRESS>', u'name': u'abc-420', u'ttl': 3600, u'type': u'A'},
{u'data': u'<IP_ADDRESS>', u'name': u'abc-421', u'ttl': 3600, u'type': u'A'}]
In [18]: client.put_a_records(domain='net.com', records=[{u'data': u'<IP_ADDRESS>', u'name': u'@', u'ttl': 600, u'type': u'A'}, {u'data': u'<IP_ADDRESS>', u'name': u'abc-420', u'ttl': 3600, u'type': u'A'},])
In [19]: client.get_a_records(domain='net.com')
Out[19]:
[{u'data': u'<IP_ADDRESS>', u'name': u'@', u'ttl': 600, u'type': u'A'},
{u'data': u'<IP_ADDRESS>', u'name': u'abc-420', u'ttl': 3600, u'type': u'A'},
{u'data': u'<IP_ADDRESS>', u'name': u'abc-421', u'ttl': 3600, u'type': u'A'}]

I'm looking into a solution to this, as currently it doesn't look like my wrapper can use godaddy's API to actually delete a domain record.  Unfortunately, GoDaddy's API is not loading for me right now and Google doesn't have the page cached.
I'm going to add this under a feature request and code it whenever the API comes back up...

Added with version 2.0.1

Will it be possible to take both hostname and ip-address/data in delete api. That way if there are two ip addresses belonging to a dns name and only one needs to be deleted then only corresponding A rec will be deleted.

I also noticed that if i add two records with different ip address.
abc.net <IP_ADDRESS>
abc.net <IP_ADDRESS>
Then only one record is seen. Ideally there should be 2 records on godaddy.
Update api should update existing record but put_a_record should insert second record ?

There is technically no "DELETE" functionality provided by the GoDaddy API.
You can implement that functionality with your own scripts now.  I have provided a new method Client.replace_records(self, domain, records, record_type=None, name=None).  You should be able to use Client.get_records perform the desired manipulations and then use Client.replace_records.  I wrote a small semblance of the delete functionality already, but I don't want to tie myself to bettering the GoDaddy API too much.  That will end up being a full-time job :P

Agree with you. Thanks a lot for providing replace_records api.
I will take a look at it.

I could't find any delete either. I ended up using a REGEX after querying the entire set of records.
So basically, query the entire set of records, check for the entry matching the one you want to delete, then if found use preg_replace to set that entry to blank (removing it) and PUT the date back.
Since I wanted the entire record (every record) for a given host name I used this REGEX:
$regex="/{[^{]+"".$hostname.""[^}]+},{0,1}/i";
Then I used that in preg_match to see if it exists and if it does then I use the same regex in preg_replace with a PUT to delete it.
Clearly, if you wanted a specific type of record you could just add that to the regex string.


Make Filter an async trait.

This PR makes the Filter trait asynchronous, allowing implementors to insert yield points. It's worth noting that you could always spawn background tasks even before this change it just wouldn't have been useful. This PR doesn't change any behaviour other adding async, but this will also allow us to try things like cancellable, concurrent, and parallel filters.
This is currently built off #293 to reduce merge conflicts, the important work is in  https://github.com/googleforgames/quilkin/commit/7c0d5fd42afae63018debed77f58f1acec4cd204
closes #12

I ran a comparison benchmark, and there is no noticeable impact between this and the main branch. So with that in mind, I would prefer that we make these async. Also with the push for GAT stabilisation async trait is getting closer to stabilisation, so I'd rather have the capability of running filters as tasks and accept the Box for now.

So I ran a quick load test with that C tool I linked earlier, I added a noop filter (simple filter that checks if the packet length is even and drops the packet if its odd, only to simulate some logic though I only sent even length packets) then ran quilkin with a config that has 10 (not sure if this is too large or too small for a typical use case) noops in the filter chain.
On my machine without async the proxy could process in the range 300-380k pps, with async it could do 260-350k pps. Its not a drastic difference but it is significant (and its larger with more filters). I mentioned earlier that the criterion benchmark we have currently is proabably too noisy to be reliable, that could be why its not detecting any difference between the two branches.
My point is not only whether or not there's a perf difference but also that its weird to add such a change at the same time that we're trying to optimize the hot path by reducing clones/allocations etc I think if there comes a need to add async filters then we should accept the async-trait Boxing (hopefully by then there's rust support for async traits :crossed_fingers: or we're in a better shape per-wise) but there isn't any such a need that we know of today? so I don't see any upside

My point is not only whether or not there's a perf difference but also that its weird to add such a change at the same time that we're trying to optimize the hot path by reducing clones/allocations etc I think if there comes a need to add async filters then we should accept the async-trait Boxing (hopefully by then there's rust support for async traits crossed_fingers or we're in a better shape per-wise) but there isn't any such a need that we know of today? so I don't see any upside

I agree with this sentiment - unless we have a feature/use case that requires this change, it makes sense to leave these kinds of changes up until we need it. No need to add extra complexity if it's not required.

I agree with this sentiment - unless we have a feature/use case that requires this change, it makes sense to leave these kinds of changes up until we need it. No need to add extra complexity if it's not required.

In #309 you say "There are lots of use cases for being able to send packets from a Filter,", that requires I/O, and if we're going to have filters performing I/O they need to use async so that we more control on how to react to filesystem failures, slow network requests, etc, and we also have #13 which is another filter which could take an indeterminate length. So I think we've already found plenty of reasons to have this feature, and have found very little cost to it.

calling e.g into an ffi isn't inherently async, the same rules for our rust filters goes for the scriptable filter I imagine.

It isn't inherently asynchronous but's a lot harder to write code that handles indeterminism when it is a function rather than a task, like I don't know how you'd write a cancellable script filter without either using async or spawning a new thread. If we chose wasmtime we could have a fuel limit in the interpreter but that would require us to always have a fuel limit on any script which I don't think would be intuitive as a default.

With #309, I don't feel like we have reached consensus there - it seems to me like there are a lot of questions unanswered. I'm not even sure that allowing a Filter to send a packet is a good idea at all -- maybe an entirely different construct is required?. With #13 - is this even a good idea? (Maybe it isn't) - what scripts would we want to support, if we did support them, what performance would we want - I have a lot of questions on both.
I'd love to see either one or the other ticket have a few "yep, that looks good to me, let's implement that" type comments before we start heading down towards implementation.

only just spotted this PR.   it certainly would have made writing my API lookup flavour of load-balancer easier (currently I'm spinning up a new OS thread for the API client and then using this trick: https://stackoverflow.com/a/66280983 to run an async fn from within the load-balancer that communicates with the API thread).
Feels to me like there are a bunch of things that could be pulled together here (e.g. #374 and #375).  From looking at the code we already have a session construct for returned traffic.  If we extended the session construct to outbound traffic then we could have a concept of a "routing" chain in addition to the filter chain, and have that chain run async and do things like load-balancing, stats export etc. that only need to happen once per session (or periodically for long-lived sessions...).
is it worth discussing this in the next community call?

Would love to hear if more people have use cases for adding the async trait to the Filter, just because of the performance hit.
So if you find one, please share!


Remove WorldLocationNewsArticle

Trello: https://trello.com/c/oNEtjIVp/99-serve-static-assets-from-www-hostname
Back in 2017 this model was replaced by a new subtype of news article,
world news story. Since then this code has lay dormant in this repo.
My motivation for removing this was relatively modest compared to the
scale of the removal. I had to make an edit to asset handling and I
noticed that world location news article would be affected, this seemed
odd since it was replaced. Then I started pulling on the string of what
could be removed and all of this came out.
Removing this has a couple of small user noticable changes:

A redirect of /government/world-location-news to
/government/announcements will go. I did some looking in CDN logs and
the only access I can find of this redirect is from me testing
what it does. It's been redirecting since 2017 so it's likely fine to
let go
Admin users can no longer choose "World Location News Article" as a
document type to filter by. Since there are 0 documents this seems to
be inconsequential.

Thanks @thomasleese. I'll merge it in and happily take full blame for any unforseen problems 🤞


Fixed escaping of '%' sign in get_type_metrics

pull request for issue #57

Hi guys,
When can we expect any action on this item?

Dear @u338steven, you are the guru of C! Could you verify the changes and drop your opinion? Thanks!

LGTM :)
@vassilevsky
https://github.com/rmagick/rmagick/blob/master/ext/RMagick/rmdraw.c
static char attrs[] = "OPbcdefghiklmnopqrstuwxyz[@#%";
% can be removed, but let's fix it some other time.
There is another issue that the number of attributes might increase.
e.g. %A, %C
It is necessary to add them to attrs[] soon.
About attributes:
Single Letter Attribute Percent Escapes:
...
%%   a percent sign
%b   file size of image read in
%c   comment meta-data property
...

Long Form Attribute Percent Escapes:
%[base]                 base filename, no suffixes (as %t)
%[channels]             ??? channels in use - colorspace ???
%[colorspace]           Colorspace of Image Data (excluding transparency)
...

ImageMagick: Format and Print Image Properties - http://www.imagemagick.org/script/escape.php

WARNING: short form percent escapes are NOT performed when the percent is after a number. For example, 10%x10 does not expand the %x as a percent escape. If you specifically want to expand the 'x', use the long form which overrides this special case. EG: 10%[x]10.

Oh... this is not considered on rmagick...

Sorry I completely forgot to merge this...
Thanks for your contribution!


Every Workspace is public regardless of what you configure

Description
If you go to {structurizrUrl}/share/{workspaceId} then every diagram is just public regardless of what is present in the DSL. I hope this is not intended.
Steps to reproduce

Spin up the container
Create a workspace
Upload a DSL with visibility private
Go to localhost:8080/share/1 and see every diagram

Screenshot
No response
Code sample
workspace {

    model {
        user = person "User"
        softwareSystem = softwareSystem "Software System" {
            webapp = container "Web Application" {
                user -> this "Uses"
            }
            container "Database" {
                webapp -> this "Reads from and writes to"
            }
        }
    }

    views {
        systemContext softwareSystem {
            include *
            autolayout lr
        }

        container softwareSystem {
            include *
            autolayout lr
        }

        theme default
    }

    configuration {
        visibility private
    }

}

Configuration
Build 2024.03.03
Severity
Critical
Priority
I have no budget and there's no rush, please fix this for free
More information
No response

Okay I got it working. You have to specify users if you want to have it private. I think the doc should maybe include that information.


Error in running the simulation

Hi,
I could generate the data using this command:
python generate_dataset.py
But when I want to run this command:
--simulation charged
It gives me this error:
error: '--simulation' is not recognized as an internal or external command, operable program or batch file.

@nzarnaghi That's an argument. Run python generate_dataset.py --simulation charged


FR: Query vault with GPT3 using embeddings

Query the user's vault with GPT3 by first gathering similar embeddings and passing as part of the prompt.

this would be a great feature and is definitely on the roadmap!


:sparkles: import data from isheep database

Description

This PR is require to include isheep data into SMARTER-database
Related Issue

closes #47
Motivation and Context

This PR is required to include isheep data into smarter database. Moreover is the first attempt to merge WGS data into a chip-based dataset
How Has This Been Tested?

Code has been tested with CI and by loading data into the SMARTER-database
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Refactoring (non-breaking change which improve code itself)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Coverage increased (+0.1%) to 92.425% when pulling 14195d474e906de8fc6ab4350e1708626e672929 on issue-47 into 6dee71df39ccf3f1e786df0303d55255ecee877b on devel.


theme: showVisitedLinks not working for some theme variants

Discussed in https://github.com/McShelby/hugo-theme-relearn/discussions/517

Originally posted by skiboyjcmu May  3, 2023
my team really likes the showVisitedLinks feature in the menu to help indicate to participants that they've completed (or at least visited) a task/chapter.  Having recently migrated from Learn to ReLearn theme, I'm finding the check marks aren't showing up in certain Themes.  You can replicate this with the reLearn theme page and the variant switcher at the bottom of nav menu
Checkmarks show up on the Relearn Light | Brighti | Neon, but not on any of the other variants.  I haven't been able to identify why it's behaving this way.  Any ideas?

Currently only light, bright and neon seem to work. Really puzzling

Currently only light, bright and neon seem to work. Really puzzling

Actually, I just realized that the checkmarks DO appear for past VisitedLinks which are not the currently selected chapter.  Also I've seen them show up briefly when switching themes, only to be immediately painted over (not sure what the technical term is, but it's as if they are there in a color, and then just being overwritten by the black background).
So I've visited all of chapter 1 & 2, and some of 3.
When I'm in a chapter 2 page, the chapter 1&3 visited checks show up:

When I'm in a Chapter 1 page, the chapter 1&3 visited checks show up:


Creating and Loading the profiles

Hello all,
I am having trouble with creating a profile via code in python. I tried creating profile in the EmotivBci and then loading it in the project, but it keeps giving me the error of missing parameters or invalid parameters.
I have followed the documentation of cortex 2.0 completely.
def setup_profile(self,name):
create_request = {
"id": 1,
"jsonrpc": "2.0",
"method": "setupProfile",
"params": {
"cortexToken": self.auth,
"profile": name,
'status': 'create'
}}
self.ws.send(json.dumps(create_request))
result = self.ws.recv()
result_dic = json.loads(result)
print('setting up profile', json.dumps(result_dic, indent=4))
def load_profile(self):
load_request = {
"id": 1,
"jsonrpc": "2.0",
"method": "setupProfile",
"params": {
"cortexToken": self.auth,
'headset_id' : self.headset_id,
"profile": "Aniruddha",
'status' : 'load'
}}
self.ws.send(json.dumps(load_request))
result = self.ws.recv()
result_dic = json.loads(result)
print('loading up profile', json.dumps(result_dic, indent=4))
I am using this in a GUI application. I am able to subscribe to 'com' for mental commands but getting stuck with loading the profile even though I have trained the profile in the EmotivBCI.
I would really appreciate it if someone could pinpoint my mistake here.

@Aniruddha008 , when call "load profile". you need to make sure your headset is connected and set correct headset serial in the param "headset"

@Aniruddha008 , when call "load profile". you need to make sure your headset is connected and set correct headset serial in the param "headset"

yes, I realized it later, that I was using 'headset_id' instead of 'headset'. Thank you so much.


curriculo-vitor

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: VitorLorente sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


MobileCoreServices has been renamed. Use CoreServices instead.

After upgrading to Xcode 11.4 I have new project warnings from Apptentive.
MobileCoreServices has been renamed. Use CoreServices instead.

Thanks for reaching out @colinrf. Taking a look at this with my team.
We'll be in touch as soon as possible.

@colinrf thanks again for the report!
We've tried to repro it using Xcode 11.4 but are not able to. Can you help use with a bit more information?

What app are you working on? If you'd prefer, please email<EMAIL_ADDRESS>with details so we can help coordinate.
Is this blocking you from release, or urgent in any other way?
How are you integrating Apptentive with your app? Any additional details that will help us repro this would be helpful.

Apptentive has been giving us the same warning in our project (we have it installed via cocoapods).  Looks like the linked PR should fix it.  Just checking to see if you'll be merging it anytime soon - we prefer to keep our project warning-free if possible.  Thanks!

Hi @chrismcs ! Thanks for letting us know. We do not have this on our near term roadmap but will let you know if that changes.
Would you mind emailing us at<EMAIL_ADDRESS>and tell us which app you're working on? I'd like to keep your Account Manager in the loop so they can help track. Just mention that you reached out on GitHub.
Last we tried, we were unable to repro this in our own tests. It sounds like you integrated via cocoapods. If there are any other details that could help us repro, let us know.
Thanks again.

Hi @CaseyApptentive, I'm also seeing this warning with Xcode 11.5 and Apptentive 5.2.11. We're using it for a hight profile client and would like to remove this warning. I'm emailing support with app details. Thanks!


fix: upgrade to axum 0.7

Description
Updates to latest version of utils-rs and thus axum 0.7 and http/hyper v1 deps.
Remaining work:

[ ] Fix header for prometheus-http-query
[ ] Consider re-using global reqwest client

How Has This Been Tested?
Existing tests?
Due Diligence

[ ] Breaking change
[ ] Requires a documentation update
[ ] Requires a e2e/integration test update

Nice 🙌 I added this to my backlog as well to update the Axum to the latest. Thanks for doing this 🙏

Thanks buddy


Runtime Error: no support for undefined or NaN values

The following error message when variable in the dataset are undefined:
    net = ml.read(main_path+"/data/"+dataset_name+"/"+dataset_name+"_graph.txt")
RuntimeError: Format error: Error converting string to double: NaN

Do you have an input file to replicate it?

I pushed an extension to development. This feature should have been added, I tried it on an input file with NULL specified where values are missing and I could retrieved None in Python.


Unable to verify mobile number

Context
When clicking on "Verify" after entering a mobile number, an error returning - "statusCode:".401"  prevents the verification process from completion.

I get the same error as @bitcoinbrisbane when trying to verify my mobile (also on iOS)

@darrenrogan  it's woking now, pls download the latest version from testflight.

Need to add instructions to enter credentials into readme

@Mitchell139 @C00p3rb123  Technically should not using local build or running locally to test, same as other projects testing on UAT, I will add instruction how to get latest uat build into readme.

@xiangzhe0812 I don't need instructions on how to test it. I moved this back because there aren't any instructions in the readme that the dev/whoever is trying to run the app has to go to some random file and enter in the message bird credentials

@Mitchell139 That is what Im trying to explain to you, you shouldn't run the app locally to test, all of the creds builded in whe ran the new build, you and @C00p3rb123  just need download/update the app from testflight or expo dev(for android) and test.


Issue with TensorFlow Federated Execution in Remote Environments

Environment:

OS: Ubuntu 22.04
Python: 3.10.0
TensorFlow: 2.14.1
TensorFlow Estimator: 2.14.0
TensorFlow Federated: 0.79.0
TensorFlow IO GCS Filesystem: 0.37.0
TensorFlow Model Optimization: 0.7.5
TensorFlow Privacy: 0.9.0
TensorFlow Probability: 0.22.1

Description:
I am encountering an issue where the following code executes successfully on my machine's desktop terminal but hangs indefinitely when run in VSCode over SSH or within a Jupyter Notebook:
tff.federated_computation(lambda: 'Hello, World!')()

Steps to Reproduce:

Run the code in the desktop terminal - it works as expected.
Run the same code in VSCode over SSH or Jupyter Notebook - the code hangs indefinitely.

Additional Information:
The code is running on CPU only.

tff.backends.native.set_local_python_execution_context() is deprecated in TFF version 0.79.
tff.backends.native.set_async_local_cpp_execution_context() does not resolve the issue.

I have referenced a similar issue described in GitHub Issue #3624.
I would greatly appreciate any guidance or solutions to help resolve this problem.
Thank you.

Ok so you have tried, https://www.tensorflow.org/federated/api_docs/python/tff/backends/native/set_async_local_cpp_execution_context.
What version of TFF are you using and on what OS?

I actually drafted an email to you, but I didn't send it. I think Github Issues might be a better approach.
Here is the TFF related packages' versions:

tensorboard==2.14.1
tensorboard-data-server==0.7.2
tensorflow==2.14.1
tensorflow-estimator==2.14.0
tensorflow-io-gcs-filesystem==0.37.0
tensorflow-model-optimization==0.7.5
tensorflow-probability==0.22.1
tensorflow_federated==0.79.0
tensorflow_privacy==0.9.0

Running on Ubuntu 22.04 under conda environment, here is the relesae detail:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Executing on the machine's terminal works, but it gets stuck in a wait state with remote connected terminal

So it sounds like you are:

SSH into a machine running Ubuntu 22.04.
Launching a Python script that uses the latest version of TFF 0.79.0.
The script invokes computations using the tff.backends.native.set_async_local_cpp_execution_context() backend.

I'm going to assume that you are using some Python version between 3.9 and 3.11?
I don't think we have a test that covers this scenario. Typically we would invoke the Python script locally and use a remote execution context that would connect to the workers remotely. However, because of issues with TF and pybind11 our C++ stack is not being built (https://github.com/google-parfait/tensorflow-federated/blob/main/tensorflow_federated/python/core/backends/native/BUILD#L62) or included in the Python package (https://www.tensorflow.org/federated/api_docs/python/tff/backends/native).
One option would be for you to construct a remote execution context. However, before I point you in that direction I vaguely remember a similar issue we had with colab notebooks. Let me try and do a bit of research on my end and get back to you.

One more thing, I just found if I use sudo or root user to execute the python script via ssh, the problem seems disappear.... But the code still can't be operated in jupyter notebook as we won't use sudo to execute code section... Very strange

Can you try and debug the notebook using these steps, https://github.com/google-parfait/tensorflow-federated/issues/4588#issuecomment-2069457726. This will give us more information and help debug the issue.

I start the binary directly with the following code section:
import subprocess
import portpicker

port = portpicker.pick_unused_port()
binary = '/home/ubuntu/miniconda3/envs/fed-py310/lib/python3.10/site-packages/tensorflow_federated/data/worker_binary'
subprocess.check_output(args=[binary, f'--port={port}'], stderr=subprocess.STDOUT)

The execution keeps on running for serveral minutes until i interrupt it. There is no any error signal during running. The following is my cpu model.
!cat /proc/cpuinfo

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 186
model name	: Genuine Intel(R) 0000
stepping	: 2

I am using Q1HY which is the Engineering Sample of Intel i9-13900HK using the structure of intel 12gen with 13gen semiconductor device fabrication. Only support AVX-2. If you need me to test on AVX-512, I'd love to setup a new machine. I also have some RTX GPUs and serveral AMD EPYC CPUs. Tell me if you need more debug info,

@ZacharyGarrett WDYT?

Still unclear what is going wrong, sounds the the child process is starting successfully which is a good sign. One issue seen in the past was that the machine didn't have a HOSTS entry for localhost (which is set here: https://github.com/google-parfait/tensorflow-federated/blob/eb0a2c378d8f22feb145ed98230de41af606810f/tensorflow_federated/python/core/impl/executor_stacks/executor_factory.py#L140)
Two things to try to gather more debug information:

Set the following in a colab cell and see what debug messages are printed:

from absl import logging
logging.set_verbosity(logging.DEBUG)
import tensorflow_federated as tff

Add two commandline flags, --logtostderr and --v=2, to subprocess.POpen call and then pipe those logs back and see what the C++ subprocess is printing.

I executed the following python script
import collections

import numpy as np
import tensorflow as tf
from absl import logging
logging.set_verbosity(logging.DEBUG)
import tensorflow_federated as tff
np.random.seed(0)

out = tff.federated_computation(lambda: 'Hello, World!')()

This output is without tff.backends.native.set_async_local_cpp_execution_context()
DEBUG:absl:Found a worker binary at: /home/ubuntu/miniconda3/envs/fed-py310/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executor_stacks/../../../../data/worker_binary
DEBUG:absl:(449489746) Entering ExecutionContext.Compile
DEBUG:absl:Compiling TFF computation to CDF.
DEBUG:absl:(913223482) Entering transform_to_native_form.to_call_dominant
DEBUG:absl:(913223482) Exiting transform_to_native_form.to_call_dominant. Elapsed time 0.000133
DEBUG:absl:Computation compiled to:
DEBUG:absl:( -> b'b"b\'Hello, World!\'"')
DEBUG:absl:(658873878) Entering transform_to_native_form.optimize_tf_graphs
DEBUG:absl:(658873878) Exiting transform_to_native_form.optimize_tf_graphs. Elapsed time 0.000024
DEBUG:absl:(610059018) Entering transform_to_native_form.transform_tf_call_ops_disable_grappler
DEBUG:absl:(610059018) Exiting transform_to_native_form.transform_tf_call_ops_disable_grappler. Elapsed time 0.000136
DEBUG:absl:(582069703) Entering transform_to_native_form.transform_tf_add_ids
DEBUG:absl:(582069703) Exiting transform_to_native_form.transform_tf_add_ids. Elapsed time 0.000021
DEBUG:absl:(449489746) Exiting ExecutionContext.Compile. Elapsed time 0.000895
DEBUG:absl:(619222267) Entering ExecutionContext.Invoke
DEBUG:absl:Starting TFF C++ server on port: 39019
DEBUG:absl:TFF service manager sleeping; stub is not ready.
DEBUG:absl:TFF service manager sleeping; stub is not ready.
DEBUG:absl:TFF service manager sleeping; stub is not ready.
DEBUG:absl:TFF service manager sleeping; stub is not ready.
DEBUG:absl:TFF service manager sleeping; stub is not ready.
DEBUG:absl:TFF service manager sleeping; stub is not ready.
DEBUG:absl:TFF service manager sleeping; stub is not ready.

With tff.backends.native.set_async_local_cpp_execution_context(), the output changes into:
DEBUG:absl:Found a worker binary at: /home/ubuntu/miniconda3/envs/fed-py310/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executor_stacks/../../../../data/worker_binary
DEBUG:absl:Found a worker binary at: /home/ubuntu/miniconda3/envs/fed-py310/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executor_stacks/../../../../data/worker_binary
DEBUG:absl:Found a worker binary at: /home/ubuntu/miniconda3/envs/fed-py310/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executor_stacks/../../../../data/worker_binary
sys:1: RuntimeWarning: coroutine 'AsyncExecutionContext.invoke' was never awaited

If i execute the following python script with sudo. The output can be shown.
sudo /home/ubuntu/miniconda3/envs/fed-py310/bin/python /home/ubuntu/projects/federated-learning/demo.py
import collections

import numpy as np
import tensorflow as tf
from absl import logging
logging.set_verbosity(logging.DEBUG)
import tensorflow_federated as tff
np.random.seed(0)

out = tff.federated_computation(lambda: 'Hello, World!')()
print(out)

DEBUG info:
DEBUG:absl:(66960220) Entering ExecutionContext.Compile
DEBUG:absl:Compiling TFF computation to CDF.
DEBUG:absl:(864885914) Entering transform_to_native_form.to_call_dominant
DEBUG:absl:(864885914) Exiting transform_to_native_form.to_call_dominant. Elapsed time 0.000137
DEBUG:absl:Computation compiled to:
DEBUG:absl:( -> b'b"b\'Hello, World!\'"')
DEBUG:absl:(995270596) Entering transform_to_native_form.optimize_tf_graphs
DEBUG:absl:(995270596) Exiting transform_to_native_form.optimize_tf_graphs. Elapsed time 0.000022
DEBUG:absl:(790988177) Entering transform_to_native_form.transform_tf_call_ops_disable_grappler
DEBUG:absl:(790988177) Exiting transform_to_native_form.transform_tf_call_ops_disable_grappler. Elapsed time 0.000101
DEBUG:absl:(688733641) Entering transform_to_native_form.transform_tf_add_ids
DEBUG:absl:(688733641) Exiting transform_to_native_form.transform_tf_add_ids. Elapsed time 0.000038
DEBUG:absl:(66960220) Exiting ExecutionContext.Compile. Elapsed time 0.000757
DEBUG:absl:(932610948) Entering ExecutionContext.Invoke
DEBUG:absl:Starting TFF C++ server on port: 35903
DEBUG:absl:TFF service manager sleeping; stub is not ready.
DEBUG:absl:TFF service manager sleeping; stub is not ready.
DEBUG:absl:Creating new ExecutorStub
DEBUG:absl:(45979872) Entering RemoteExecutor.set_cardinalities
DEBUG:absl:(168720986) Entering remote_executor_grpc_stub._request
DEBUG:absl:(168720986) Exiting remote_executor_grpc_stub._request. Elapsed time 0.001591
DEBUG:absl:(45979872) Exiting RemoteExecutor.set_cardinalities. Elapsed time 0.001677
DEBUG:absl:(870858287) Entering RemoteExecutor.create_value
DEBUG:absl:(543758160) Entering <locals>.serialize_value
DEBUG:absl:(555035122) Entering value_serialization.serialize_value
DEBUG:absl:(452438478) Entering value_serialization._serialize_computation
DEBUG:absl:(452438478) Exiting value_serialization._serialize_computation. Elapsed time 0.000086
DEBUG:absl:(555035122) Exiting value_serialization.serialize_value. Elapsed time 0.000148
DEBUG:absl:(543758160) Exiting <locals>.serialize_value. Elapsed time 0.000217
DEBUG:absl:(690390915) Entering remote_executor_grpc_stub._request
DEBUG:absl:(690390915) Exiting remote_executor_grpc_stub._request. Elapsed time 0.000384
DEBUG:absl:(870858287) Exiting RemoteExecutor.create_value. Elapsed time 0.000710
DEBUG:absl:(1038647) Entering RemoteExecutor.create_call
DEBUG:absl:(408776496) Entering remote_executor_grpc_stub._request
DEBUG:absl:(408776496) Exiting remote_executor_grpc_stub._request. Elapsed time 0.000217
DEBUG:absl:(1038647) Exiting RemoteExecutor.create_call. Elapsed time 0.000281
DEBUG:absl:(453475155) Entering RemoteValue.compute
DEBUG:absl:(722504673) Entering RemoteExecutor._compute
DEBUG:absl:(161901310) Entering remote_executor_grpc_stub._request
DEBUG:absl:(161901310) Exiting remote_executor_grpc_stub._request. Elapsed time 0.001115
DEBUG:absl:(382997197) Entering value_serialization.deserialize_value
DEBUG:absl:(3427812) Entering value_serialization._deserialize_computation
DEBUG:absl:(3427812) Exiting value_serialization._deserialize_computation. Elapsed time 0.000062
DEBUG:absl:(382997197) Exiting value_serialization.deserialize_value. Elapsed time 0.000110
DEBUG:absl:(722504673) Exiting RemoteExecutor._compute. Elapsed time 0.001299
DEBUG:absl:(453475155) Exiting RemoteValue.compute. Elapsed time 0.001336
DEBUG:absl:(932610948) Exiting ExecutionContext.Invoke. Elapsed time 2.009607
b'b"b\'Hello, World!\'"'


Add JSONP Compatibility

Currently, a JSON object is sent in response to an HTTP get request. JSONP can be supported by adding an optional callback function.

New projects will use JSONP moving forward.


LOAD breaks silently if other data is present in the ROM section

SECTION "Foo", ROM0
    db 1, 2
LOAD "wFoo", WRAM0
    db 3
ENDL

SECTION "Bar", ROM0
    db 4

~/rgbds% ./rgbasm test.asm -o test.o
~/rgbds% ./rgblink test.o -o test.gb -m test.map
~/rgbds% hexdump -C test.gb
00000000  01 02 04 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00004000
~/rgbds% cat test.map
ROM0 bank #0:
  SECTION: $0000-$0001 ($0002 bytes) ["Foo"]
  SECTION: $0002-$0002 ($0001 byte) ["Bar"]
    SLACK: $3ffd bytes

WRAM0 bank #0:
  SECTION: $c000-$c000 ($0001 byte) ["wFoo"]
    SLACK: $0fff bytes

As you can see, the LOAD data gets silently overwritten by other sections.

Fixed in 9d62b4b.


Render SVG using this package via JS

Hello,
Is it possible to use this package and render SVG using JS?

Heya, no this is a PHP only package.


Translate file "blade.md"

Notes for contributors, the file can be translated by being aware of and understanding the file you are translating, you must have an understanding of the techniques and the explanation provided by the documentation file, and it must be in Standard Arabic, and the technical terms should be translated as follows:
For example, المتحكمات with the term next to it in English (Controllers)
For example, التهجير and next to it the term in English (Migrations)
And so on
Thank you for your contribution

I will translate blade.md file

Of course, thanks


Important Updates to CNCopyCurrentnetworkInfo API in iOS 13

The following email recently went out to Apple developers concerning new network privacy restrictions starting in iOS 13 which I believe will affect wifi setup in checkParticleDeviceWifiConnection and encourages migration to NEHotSpotConfiguration introduced in iOS 11.
https://github.com/particle-iot/particle-setup-ios/blob/c01471328ac1260d207f42b86fe0a6de608dd7a2/ParticleSetup/Comm/ParticleSetupCommManager.m#L128

Dear Developer,

As we announced at WWDC19, we're making changes to further protect user privacy and prevent unauthorized location tracking. Starting with iOS 13, the CNCopyCurrentNetworkInfo API will no longer return valid Wi-Fi SSID and BSSID information. Instead, the information returned by default will be:

SSID: “Wi-Fi” or “WLAN” (“WLAN" will be returned for the China SKU)
BSSID: "00:00:00:00:00:00"

If your app is using this API, we encourage you to adopt alternative approaches that don’t require Wi-Fi or network information. Valid SSID and BSSID information from CNCopyCurrentNetworkInfo will still be provided to VPN apps, apps that have used NEHotspotConfiguration to configure the current Wi-Fi network, and apps that have obtained permission to access user location through Location Services.

Test your app on the latest iOS 13 beta to make sure it works properly. If your app requires valid Wi-Fi SSID and BSSID information to function, you can do the following:

For accessory setup apps, use the NEHotSpotConfiguration API, which now has the option to pass a prefix of the SSID hotspot your app expects to connect to.
For other types of apps, use the CoreLocation API to request the user’s consent to access location information.

Learn more by reading the updated documentation or viewing the the Advances in Networking session video from WWDC19. You can also submit a TSI for code-level support.

Best regards,
Apple Developer Relations

Hello, thank you for creating an issue. I have received the same email too and will look into this next week. As a quick workaround there's an option to request location permission. Will keep this thread updated with the progress.

Latest release (https://github.com/particle-iot/particle-setup-ios/releases/tag/0.9.1) addresses the issue. If iOS 13 is detected, user is presented with a screen explaining why we need location permission to advance. Please also note that you will need to provide Privacy - Location When In Use Usage Description in info.plist.

Thanks very much for addressing that @RaimundasSakalauskas 👍
Are there any plans to move to NEHotspotConfiguration ? From my understanding it

Doesn't require the user to leave the app at all, making the setup process much simpler for our users,
Is available from iOS 11, which actually includes our whole user base at present, and
Seems to be fairly simple to implement from a code perpective using joinOnce if the SSID is known? ref https://developer.apple.com/documentation/networkextension/nehotspotconfiguration/2887518-joinonce?language=objc & https://stackoverflow.com/a/46078618/1277350

I am the loudest ambassador of us switching to NEHotspotConfiguration, however to do that, we need to change both - our manufacturing process and setup process. That API requires us to know exact SSID beforehand, which app does not right now. I am trying to figure out if it's possible to get SSID if user scans serial number barcode located on the box. Even if that is possible, current gen of Photons don't have serial numbers on them (only on the box), so unless SN is added on the hardware in the future, current setup process would still have to be there for all existing photons.

Thanks very much for explaining @RaimundasSakalauskas 👍
I see what you mean, the prefix is configurable but the exact SSID isn't necessarily known:
https://github.com/particle-iot/particle-setup-ios/blob/c01471328ac1260d207f42b86fe0a6de608dd7a2/ParticleSetup/Comm/ParticleSetupCommManager.m#L136
Much appreciated!


ci: [2024-Q3] CI/CD Audit Story

Contents

CI/CD Repository Audit

Contents
Administrative Audit Criteria

Check Actions State
Check if Actions should be disabled
Repository Settings Checks
App Integrations
Security Checks
Custom Properties

Non-Administrative Audit Criteria

Dependabot
Workflow checks
Self Hosted Runners
CODEOWNERS
Other

Repository Settings
Acceptance Criteria

Administrative Audit Criteria
Check Actions State

[ ] Actions are enabled
[ ] Actions are disabled

Check if Actions should be disabled
If actions have not been run in the previous 6 months they should be disabled:

[ ] Actions have run in the last 6 months and shall remain enabled
[ ] Actions have been disabled on the inactive repository

Repository Settings Checks

[ ] Repository settings are configured per organization standard
[ ] Individual branch protections are turned off
[ ] Individual tag protections are turned off
[ ] The repository uses the current rulesets
[ ] Teams are assigned to the repository
[ ] Individual contributors that are part of assigned teams are removed from contributors list
[ ] All webhooks present are needed and in use

App Integrations
If actions are enabled:

[ ] Dependabot is enabled on the repository
[ ] Codecov is enabled on the repository

Security Checks

[ ] Snyk is enabled on the repository
[ ] Dependabot is configured to monitor all relevant ecosystems

npm
electron
github actions
etc.

[ ] Secrets Management

[ ] No hardcoded secrets in the workflow files or code
[ ] GitHub secrets are employed to store sensitive data
[ ] Secrets are referenced in CI via config files or environment variables

[ ] Tokens are stored securely as GitHub Secrets
[ ] Executable Path Integrity

[ ] Integrity checks for executables are implemented

integrity checks should use either checksums or cryptographic hashes for verification

[ ] Checksums/hashes are verified during CI process to detect unathorized changes
[ ] Expected checksums/hashes are stored securely and referenced through the CI pipeline

[ ] Code Coverage Reporting - Configure codecov on the repository
[ ] CodeQL is enabled on the repository
[ ] npx playwright install deps is used to install OS dependencies instead of aptitude
[ ] Code Formatting

[ ] ESLint rules are applied to the codebase
[ ] Prettier Formatting rules are applied to the codebase

Custom Properties

[ ] Custom properties: last-ci-review-by-team is set
[ ] Custom properties: last-ci-review-date is set (Use format: YYYY-MM-DD)

Non-Administrative Audit Criteria
Dependabot

[x] dependabot.yml is up to date

Workflow checks

[x] Appropriate permissions are set within the github workflows
[x] All steps are named
[x] All workflow actions are using pinned commits
[x] The Step-Security Hardened Security action is enabled on each workflow job
[x] Ensure no hard-coded keys in workflows

[ ] Alert devops-ci administrative team if new github secrets are needed to resolve hard-coded keys

Self Hosted Runners

[x] The Repository is using the latitude runner group label for the runs-on stanza

CODEOWNERS

[x] .github/CODEOWNERS is valid and up-to-date

Other

[ ] If Applicable: Alert repository owners of software versions that are no longer supported
[ ] If Applicable: Alert repository owners when software versions are within 3 months of losing support

Repository Settings

[x] Require contributors to sign off on web-based commits
[x] Features: Issues
[x] Features: Preserve this Repository
[x] Features: Discussions
[x] Features: Projects
[x] Pull Requests: Allow Squash Merging
[x] Pull Requests: Always suggest updating pull request branches
[x] Pull Requests: Automatically delete head branches
[x] Pushes: Limit how many branches and tags can be updated in a single push

Acceptance Criteria

[ ] All Audit Criteria have been met

Non-administrative checks are done. @andrewb1269hg assigning over to you.

Removed AlfredoG87, david-bakin-sl, ebadiere, quiet-node from the contributor list as they're included in the hedera-smart-contracts team.
Removed georgi-l95, Ivo-Yankov, konstantinabl, MiroslavGatsanoga, natanasow from the contributor list as they're included in the limechain-hedera team.

Administrative items are cleaned up and complete.
Over to @mishomihov00 to close out PR #998 then close out this issue.


Release 0.27.0 archive won't build while checkout tag does

A source download from the Releases page fails to make but checking out tag v0.27.0 succeeds. (MacOS 10.11.5, OCaml 4.03.0)
To repro,
wget https://github.com/facebook/flow/archive/v0.27.0.tar.gz
open v0.27.0.tar.gz && cd flow-0.27.0/
make

It stops after,
mkdir -p _build/src/embedded/
cp src/embedded/flowlib_elf.c _build/src/embedded/flowlib_elf.c
cd _build/src/embedded/ && ocamlopt   -ccopt -DNO_LZ4 -c flowlib_elf.c
ocaml -I scripts -w -3 unix.cma scripts/gen_build_id.ml _build/hack/utils/get_build_id.gen.c

This causes https://github.com/Homebrew/homebrew-core/pull/1883 to fail, requiring a checkout https://github.com/Homebrew/homebrew-core/pull/2196#issuecomment-227196507.

i think this is the same underlying issue as https://github.com/facebook/flow/issues/1972: if hg and/or git aren't installed, gen_build_id.ml seems to stall forever.
when you check out the tag, are you running it any differently? like maybe in a container that has git vs one that doesn't?

By container you mean Docker? I'm just running this in MacOS terminal. Same shell env. The above doesn't work and the following does,
git clone git@github.com:facebook/flow.git
cd flow
git checkout v0.27.0
make

Ending with,
mkdir -p bin
cp _build/src/flow.native bin/flow

ok i can repro by removing hg from my path, even when git is installed. not sure why it works from a git checkout... still looking into it.

fix incoming


Generating synthetic localizations package failed with 1 error:  Exception: The 'template-arb-file', LocalFile: 'E:\FlutterDevelopment\pro\holiday\lib/l10n\app_en.arb', is not readable. Please ensure that the user has read permissions.

Steps to Reproduce

E:\FlutterDevelopment\pro\holiday>flutter run
Generating synthetic localizations package failed with 1 error:
Exception: The 'template-arb-file', LocalFile: 'E:\FlutterDevelopment\pro\holiday\lib/l10n\app_en.arb', is not readable.
Please ensure that the user has read permissions.
l10n.yaml
arb-dir: lib/l10n
template-arb-file: app_en.arb
output-localization-file: app_localizations.dart

Unable to run the app and generate localizations file . I have follow the sample doc  https://docs.flutter.dev/development/accessibility-and-localization/internationalization

Hi @rddewan,  Thanks for filing the issue. Please share your flutter doctor -v and the complete error logs. The issue seems to be related to permissions please also try changing the location of your project.
Thanks

@maheshmnj  Below is the log file. I have full permission on the file path it's located E:\FlutterDevelopment\pro\holiday
E:\FlutterDevelopment\pro\holiday>flutter doctor -v
╔════════════════════════════════════════════════════════════════════════════╗
║ A new version of Flutter is available!                                     ║
║                                                                            ║
║ To update to the latest version, run "flutter upgrade".                    ║
╚════════════════════════════════════════════════════════════════════════════╝
[√] Flutter (Channel stable, 2.5.2, on Microsoft Windows [Version 10.0.19042.1348], locale en-US)
• Flutter version 2.5.2 at C:\flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 3595343e20 (8 weeks ago), 2021-09-30 12:58:18 -0700
• Engine revision 6ac856380f
• Dart version 2.14.3
[!] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
• Android SDK at E:\AppData\Android\sdk
X cmdline-tools component is missing
Run path/to/sdkmanager --install "cmdline-tools;latest"
See https://developer.android.com/studio/command-line for more details.
X Android license status unknown.
Run flutter doctor --android-licenses to accept the SDK licenses.
See https://flutter.dev/docs/get-started/install/windows#android-setup for more details.
[√] Chrome - develop for the web
• Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe
[√] Android Studio (version 2020.3)
• Android Studio at C:\Program Files\Android\Android Studio
• Flutter plugin can be installed from:
https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)
[√] Android Studio (version 2020.3)
• Android Studio at C:\Program Files\Android\Android-studio-2<IP_ADDRESS>
• Flutter plugin can be installed from:
https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)
[!] Android Studio (version 4.2)
• Android Studio at C:\Program Files\Android\Android Studio 4.2 Beta5
• Flutter plugin can be installed from:
https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
https://plugins.jetbrains.com/plugin/6351-dart
X Unable to determine bundled Java version.
• Try updating or re-installing Android Studio.
[√] IntelliJ IDEA Ultimate Edition (version 2020.1)
• IntelliJ at C:\Program Files\JetBrains\IntelliJ IDEA 2020.1.1
• Flutter plugin can be installed from:
https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
https://plugins.jetbrains.com/plugin/6351-dart
[√] VS Code (version 1.62.3)
• VS Code at C:\Users\Richard\AppData\Local\Programs\Microsoft VS Code
• Flutter extension version 3.28.0
[√] Connected device (3 available)
• sdk gphone64 x86 64 (mobile) • emulator-5554 • android-x64    • Android 11 (API 30) (emulator)
• Chrome (web)                 • chrome        • web-javascript • Google Chrome 96.0.4664.45
• Edge (web)                   • edge          • web-javascript • Microsoft Edge 96.0.1054.29
! Doctor found issues in 2 categories.
E:\FlutterDevelopment\pro\holiday>

Hi @rddewan, I verified on stable 2.5.3 and I could not reproduce the issue the app does run and generate localization files. Can you please try running this sample appand see if it generated the files. If you still face the same issue with this app please try upgrading to stable 2.5.3 by running
flutter upgrade.

flutter doctor -v
[√] Flutter (Channel stable, 2.5.3, on Microsoft Windows [Version 10.0.19042.1348], locale en-US)
    • Flutter version 2.5.3 at C:\flutter_sdk\stable
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 18116933e7 (6 weeks ago), 2021-10-15 10:46:35 -0700
    • Engine revision d3ea636dc5
    • Dart version 2.14.4

[√] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at C:\Users\mahesh\AppData\Local\Android\sdk
    • Platform android-31, build-tools 31.0.0
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[√] Visual Studio - develop for Windows (Visual Studio Community 2019 16.11.4)
    • Visual Studio at C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
    • Visual Studio Community 2019 version 16.11.31727.386
    • Windows 10 SDK version 10.0.19041.0

[√] Android Studio (version 2020.3)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7249189)

[√] VS Code (version 1.61.2)
    • VS Code at C:\Users\mahesh\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.28.0

[√] Connected device (3 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.19042.1348]
    • Chrome (web)      • chrome  • web-javascript • Google Chrome 96.0.4664.45
    • Edge (web)        • edge    • web-javascript • Microsoft Edge 95.0.1020.44

• No issues found!


how to use the vector database in web?

you can use Qdrant, this library can use directly in python


rename "Lokalexperten" in produktiv instanz

In Produktivinstanz #50survivors
the contributors should be renamed as
"50survivors" instead of
"Lokalexperten"

What about "Survivors"? Oder "Überlebende"?

In #264 it's now called "Mitglieder".

@roschaefer @FrauCsu I think this isn’t necessary anymore as we now call them "Community" and "Mitglieder", right?

Yes.


Update preview TSL certificate ARN

Updates certificate name to match the new letsencryp certificate.
Letsencrypt update job will always rename the new certificate to
keep the current name, so future renewals should require no more
changes.
Removes iam_certificate_id since it appears to be unused now.

:+1:


If we change value in neo4j directly,  can we remain reactivity ?

Dear @dr-dimitru
When I was changing value in Neo4j browser, the front end did not change reactively. In my case, I want to execute python scripts to change data in Neo4j on server, which aims to remain reactivity at same time. Is there a way to achieve such kind of reactivity? And I tried meteor-mysql https://github.com/numtel/meteor-mysql . I changed the value in phpMyadmin, the front end changed reactively.
At last I want to use meteor-neo4j in this way,
Front End ------ call method ------> python script ( gensim, panas ) ---------> Neo4j --------> Front End
If we can't right now, is there another way to achieve such kind of functions?
This is the snippet of calling python.
https://bitbucket.org/drushing/themeddlinproject/wiki/Server-side Python Script Execution
Thank you very much!

Hi @wuxianliang ,
As neo4j doesn't have any hooks/watchers/observer - reactivity works only when you change data via Neo4jReactivity package.
The MySQL you have mentioned, redis and mongodb has built-in observers, that's why data was reactivily changed when you've changed the value in phpMyadmin.
Sorry for this inconvenience, but where is no way to watch data changes on neo4j and push it to subscribed channels, as it happens with mysql/redis/mongodb

Thank you. I am trying  to archieve somethings like this with python-meteor, which met problems.

Well I just watched a video: http://youtu.be/jkfnr2hxb9Y?t=1m32s and they are creating nodes and the changes spread to the client. This is the repository: https://github.com/kainolophobia/neo4j-livedata

@FranciscoGutierrez neo4j-livedata project is officially dead
Theirs watcher was based on http requests, on which neo4j is based.
Also by the fact they are poll new data - polling_observe and this isn't look like a good idea


Multiple language suppport

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 18f467d3-3b0c-aec7-4129-1670b4df4acd
Version Independent ID: f85a2bb8-ab47-24d8-c312-f1bd3bccabff
Content: Use the form processor component in Power Apps - AI Builder
Content Source: ai-builder-docs/form-processor-component-in-powerapps.md
Service: powerapps
GitHub Login: @JoeFernandezMS
Microsoft Alias: jofernan

Form processor component in Power Apps only support English language at this time, because: "The Form Processor in AI Builder uses the Form Recognizer Cognitive Service under the hood, which only supports English at this time."  This we have received this information from DEV. Lets add this information on public facing document as well.

Submitted PR to state in the requirements that the only supported language currently for form processing is English: https://github.com/MicrosoftDocs/ai-builder-pr/pull/295


fix: remove ambiguity in BatchSizeSettings with PolyesterForwardDiff

Versions

Bump DI to v0.6.19

DI extensions

PolyesterForwardDiff: define two methods for BatchSizeSettings (array and integer) instead of one. Fixes #605

@avik-pal can you try this branch to see if your problem is fixed?

Cool. I'll see what the Enzyme folks have to say about https://github.com/EnzymeAD/Enzyme.jl/issues/2056 but either way I'll merge and release this soon


Better Airlock & Meat spike forensics

About the PR

Airlocks now include fingerprint/fiber forensics when bumped into, as opposed to only when manually clicked.
Meat spikes now include DNA forensics when a creature is spiked, and while spiked it displays the creature's name.
Attaching someone to a meat spike includes fingerprints/fiber forensics.
Meat carved off a spike no longer has the person's name, but instead includes the DNA.
Carving meat gives DNA forensics on the knife used.
Why / Balance

Airlocks not including forensics when bumped created an annoying dissonance where seemingly equal actions (bumping & manually clicking) resulted in different outcomes not apparent to the user. You can still avoid leaving forensics on a door by throwing your PDA, but that's a much more deliberate action.
Meat spikes generally needed some touch-up in the forensics department; identifying meat by a name tag was a bit weird and not diagetic.
Note that cooking food does not preserve DNA data; best start frying up some burgers, Syndie-chefs!
Technical details

The PR makes BeforeDoorOpenedEvent into a virtual method to allow the Server system AirlockSystem to use it.
Also includes unknown DNA for meat carved from non-DNA holders (e.g. monkeys), to future-proof against #26699.
Media

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Changelog

:cl:

tweak: Meat spikes and carved meat can now be scanned for forensics.
remove: Carved meat no longer include the victim's name.
fix: Airlocks now include fingerprint/fiber forensics when bumped!

I don’t know if this could also be done but could blood contain the DNA of the person its from, currently blood contains nothing usable by a detective

I don’t know if this could also be done but could blood contain the DNA of the person its from, currently blood contains nothing usable by a detective

Being handled in #26699! :)

I don’t know if this could also be done but could blood contain the DNA of the person its from, currently blood contains nothing usable by a detective

Being handled in #26699! :)

:O

Closed due to feature freeze May 10th-June 14th. Comment to have it reopen after this.

Doth the freeze begoneth, I plead of thee


Gmail blocks open in a new window

Hello,
I'm experiencing this issue where every time I click on a link from Gmail it won't let me open it and a window comes up saying that popups are likely blocked. Any other service works correctly (links on Whatsapp, Telegram, etc). Any idea of how I should fix it? It's very important to be able to me to click on the links I receive by email.
Thank you!

This is now fixed by https://github.com/getferdi/ferdi/pull/1474 - which has just been merged! Please pull and rebuild from source, or wait for tonight's nightly build.


Unsoundness in dynamic queries: sparse filters are ignored if the typed query is dense

Bevy version
Both 0.14.0 and af865e76a323fc87dd16157e7f4d7593d02c2b95
What you did
#[derive(Component)]
struct Dense;

#[derive(Component)]
#[component(storage = "SparseSet")]
struct Sparse;

let mut world = World::new();

world.spawn(Dense);
world.spawn((Dense, Sparse));

// How this is not working as intended:
let mut query = QueryBuilder::<&Dense>::new(&mut world)
    .without::<Sparse>()
    .build();
// Prints 2 but only 1 entity should be matched.
dbg!(query.iter(&world).count());

// How this is unsound:
fn sys(q1: Query<&mut Dense, With<Sparse>>, q2: Query<(Entity, &mut Dense)>) {
    for (entity, _) in &q2 {
        assert!(!q1.contains(entity));
    }
}
SystemBuilder::<(Query<&mut Dense, With<Sparse>>,)>::new(&mut world)
    .builder::<Query<(Entity, &mut Dense)>>(|builder| {
        builder.without::<Sparse>();
    })
    .build(sys)
    .run((), &mut world);

What went wrong
In the simple demonstration the query contains 2 entities, but only 1 should match all the filters. The reason both are yielded is because the static part of the query is dense and so the dense iteration optimization kick in and iterates over the table matching it. But doing so completly ignores the filter on the sparse component.
In the soundness issue this is exploited in a dynamic system where the sparse filter is used to make the two queries disjoint, which is however broken by the bug just showed.

As mentioned on Discord, this is effectively #13120 again.

The difference is that for #13120 it was just one of the blockers for merging, while this issue is present on an already released version.

So the problem is that queries use static analysis on the type of the query to check if all components are dense, in which case it iterates through the table instead of the archetype?
Would the solution be to check if the query is dense by looking at the Access dynamically instead of looking at the types of D and F?

So the problem is that queries use static analysis on the type of the query to check if all components are dense, in which case it iterates through the table instead of the archetype?

Yes. Basically the (wrong) assumption is that if all the static types and filters are dense then the query will actually access all and only those entities in the corresponding tables, but that assumption is violated by the query not having access to some of the archetypes in those tables when built dynamically (or when transmute/transmute_lens/etc etc are used, see #14528).

Would the solution be to check if the query is dense by looking at the Access dynamically instead of looking at the types of D and F?

It would solve the soundness issue, but it would also require introducing a branch in every call to the very hot QueryIter::next which would likely be devastating for performance.

It would solve the soundness issue, but it would also require introducing a branch in every call to the very hot QueryIter::next which would likely be devastating for performance.

Could something like:

at QueryIter  creation time (QueryIter::new())  we create either a QueryArchetypeIterationCursor or QueryTableIterationCursor depending on the access, which is checked dynamically
then the hot path of Cursor::next() wouldn't have branching

work?

No, Cursor::next() would still have a branch due to checking which cursor it actually is. Ultimately the two types of iteration do different things, so if you want to make the selection dynamic you will have a branch in next.

No, Cursor::next() would still have a branch due to checking which cursor it actually is. Ultimately the two types of iteration do different things, so if you want to make the selection dynamic you will have a branch in next.

I wouldn't have any dynamic checks in the cursor because I have 2 different types of Cursors. DenseCursor vs ArchetypeCursor. Maybe I can try it

A solution which would keep the dense iteration optimization may be to add a new marker parameter to Query and QueryState which tells it whether it can trust the static filters or not. By default it would be to trust the static filters, and thus the dense iteration optimization would be possible, but any operation that may introduce dynamic ones will have to return a Query/QueryState with the other marker where it is not.
The downside of this approach is that now we have two incompatible kind of Query, though we could add a zero-cost way to convert a static one to a dynamic one to ease the reduction in usability.

would not work because the check of which type of cursor we are is still done dynamically?

Yes

I was hoping that after the Cursor creation, the other functions would just be called statically. We only need to check the is_dense at cursor creation, not after

In the end you want to do different things in next depending on whether the query is actually dense or not, so you need a branch there, there's no way around that. Moving the dense check at cursor creation just changes what the branch in next will be, but ultimately you need a branch or to let go of the dense iteration optimization.


Add assertions on breadcrumbs to queue tests

Goal
This PR adds assertions to the number of breadcrumbs left in the queue tests. This is important because we rely on the "looping" event to clear breadcrumbs between jobs:
https://github.com/bugsnag/bugsnag-laravel/blob/1dd9fb6a658f59f784657869d0d8028c583a59f0/src/BugsnagServiceProvider.php#L157-L161
In order to make this work consistently across Laravel versions, I've disabled the default query breadcrumbs and left breadcrumbs at various lifecycle points:

when the AppServiceProvider boots
before a job
after a job (only fires for successful jobs)
when an exception occurs in a job (not supported on Laravel 5.1)
inside both HandledJob and UnhandledJob

Laravel 5.1 doesn't support all of these events, so has different breadcrumbs than the rest of the fixtures
I've also refactored the feature file to simplify the steps where there's a difference between Laravel versions. Instead of:
And on Laravel versions >= 5.2 the event "metaData.job.name" equals "Illuminate\Queue\CallQueuedHandler@call"
And on Laravel versions >= 5.2 the event "metaData.job.queue" equals "default"
And on Laravel versions >= 5.2 the event "metaData.job.attempts" equals 1
And on Laravel versions >= 5.2 the event "metaData.job.connection" equals "database"

You can now do this:
And on Laravel versions >= 5.2:
  """
  the event "metaData.job.name" equals "Illuminate\Queue\CallQueuedHandler@call"
  the event "metaData.job.queue" equals "default"
  the event "metaData.job.attempts" equals 1
  the event "metaData.job.connection" equals
  """

I think being required to write steps as strings it would be better to write the assertions out as a table, e.g.:
Then the following assertions are true on Laravel versions > 5.2:
    | app.type    | Queue |
    | other.thing | yes      |

I realise this may seem like semantics, but we're already discussing steps having too many sub-steps included for the logic to be clear when reading the feature files. While this is more explicit about the steps being run, it still makes it more difficult to parse when reading, even if it's more flexible.

I'm not sure how to implement that for all three types of steps currently used without having to handle each separately:

the event {string} equals {string} (also "is null")
the event has a {string} breadcrumb named {string}
the event has {int} breadcrumbs

It would be good to have a better way to do this in general though — I think it's actually quite common to have differences across versions but we typically make the assertions looser to fit, e.g.:
https://github.com/bugsnag/bugsnag-laravel/blob/1dd9fb6a658f59f784657869d0d8028c583a59f0/features/unhandled_view.feature#L8-L12
https://github.com/bugsnag/bugsnag-laravel/blob/1dd9fb6a658f59f784657869d0d8028c583a59f0/features/ooms.feature#L10-L12
Having the ability to keep assertions strict by running them conditionally based on the version is really nice IMO and I think it'd be a shame to lose the flexibility of allowing any step to be conditional
It'd be much nicer to e.g. use tags on a step to make it conditional, but cucumber doesn't support that and it doesn't have a skip_this_step (like skip_this_scenario) AFAICT so this was the best I could come up with — I would be very happy to use some other syntax, e.g. cucumber allows * instead of Given/When/Then/And so this would be ideal:
Given something
Then on Laravel versions >= 5.2:
* something
* another thing
* another another thing

but I couldn't figure out how to get that working

@Cawllec how do you feel about this?
| laravel version | step                     |
|          >= 5.2 | the event "a" equals "b" |
|                 | the event "c" equals "d" |
|                 | the event "e" is null    |
|           < 5.2 | the event "a" equals "x" |
|                 | the event "c" equals "y" |

TBH I like that even less, but I don't have a proposal that will accomplish the same thing right now.  I think that we can use it for now without issue, and I'll create a ticket to review how we're using cucumber/gherkin and if we need to refactor the step layouts as a whole (based on how we want to address it)


Make GMM documentation more visible, fix bug in M step of EM algorithm

Codecov Report

Merging #144 (801f1fe) into main (cca7453) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #144      +/-   ##
==========================================
- Coverage   82.77%   82.77%   -0.01%
==========================================
  Files          51       51
  Lines        4946     4945       -1
  Branches      502      502
==========================================
- Hits         4094     4093       -1
  Misses        702      702
  Partials      150      150

Impacted Files
Coverage Δ

ott/tools/gaussian_mixture/fit_gmm_pair.py
76.82% <100.00%> (-0.28%)
:arrow_down:


does not support MATCHES w/ _bin data

If you add the following test case to test_binary.py (https://github.com/oasis-open/cti-pattern-matcher/blob/master/stix2matcher/test/test_binary.py ):
"[binary_test:name_bin MATCHES 'alice' ]",
you will get a failure, even though it should work.  This is explicitly allowed in the spec via the statement:
In the case that the property is binary (e.g., the property name ends in _bin or _hex), then the UNICODE flag MUST NOT be specified.

Hmm... I just responded to issue 39 under the assumption MATCHES only makes sense with textual data.  The pattern matcher doesn't support regex matching on binary data, because I didn't realize it needed to.  I guess I missed/misunderstood that part.
I couldn't find mention of a UNICODE flag/option (mentioned in spec section 4.2.1) on the PCRE website though.  The pattern page doesn't list it as a settable option within a regex.  The API page doesn't list it as a regex compile option either.  It's not clear to me that PCRE claims to support matching on binary data.  Seems like it considers match targets as text always, although some characters may be unprintable.  That means it's not clear what that reference to the "UNICODE flag" is really trying to convey.
But I get it, regex matching should work on binary data.  I'll have to add that.

There are the PCRE_UTF* flags on compile: http://www.pcre.org/original/doc/html/pcre_compile.html
As all data in Cybox is Unicode (but encoding not specified), it is up to the implementation to decide WHICH Unicode encoding they use and use the correct flag.  It is likely that Windows will use _UTF16, while most POSIX OS's will use _UTF8, but we cannot mandate which Unicode encoding is used in the specification.

Ok, so the _UTF* flags tell PCRE to decode what they're given (both pattern and target), to unicode code points, before performing the pattern match.  The STIX Pattern spec is saying don't do that (to observable values.  Patterns are always strings).

So, strings in cybox are always Unicode.  There is no encoding specified except that JSON has a default of UTF-8, but there can be JSON in UTF-16 too.   As the strings are Unicode, there is no decoding to be done, but you have to encode the unicode strings to match the UTF flag you pass to PCRE (or the python PCRE library should handle this automatically for you).

As all data in Cybox is Unicode...

Actually, section 2 of spec part 3 (Cyber Observable Core Concepts) lists 13 data types. :)  This quote is from three boxes up, but perhaps your intended meaning was more complex: All binary data in Cybox, when serialized as JSON, becomes a unicode string.  Maybe imprecise language is getting us into trouble.

So, strings in cybox are always Unicode.

Yes, but the point was to apply patterns to instances of the cybox binary and hex types (which can collectively be thought of as "binary data").  So I understand the directive in the STIX Pattern spec to say that in the context of a match operation, if you encounter binary data in an observable, don't try to decode to unicode (it wouldn't be clear how to anyway, since there isn't generally any way to know what the encoding is).  If matching on binary data, you must interpret the regex as a "binary regex" and use the data directly.
To my mind, the implications of a "binary regex" are that the regex, which is required to be a string literal, must be convertible to binary as defined in section 4.2.4 (no code points > 255).  Higher numbered code points could never match to a byte, so it wouldn't make sense.  Also, many regex features require textual semantics.  When applying to sequences of bytes, they don't make sense.  E.g. what would the regex \d (match a numeric digit) mean when applied to binary data?  You'd have to ascribe textual semantics to various byte values, e.g. use ASCII interpretation.  So \d would match byte values in the range 48-57, inclusive.
So there are deeper considerations implied by using regexes to match binary data, which aren't spelled out in the spec, and aren't evident in the reference to the PCRE "UNICODE" flag (which doesn't seem to even exist).  The above is the way I expect my pattern matcher implementation to work since that's how Python's regexes work as far as I know, and nothing in the spec says it can't work that way.
I don't see details of JSON serializations as particularly relevant to this; all specs should be written in a serialization-neutral manner (although perhaps with an eye to making particular serializations straightforward and usable).

As all data in Cybox is Unicode...
Actually, section 2 of spec part 3 (Cyber Observable Core Concepts) lists 13 data types. :) This quote is from three boxes up, but perhaps your intended meaning was more complex: All binary data in Cybox, when serialized as JSON, becomes a unicode string. Maybe imprecise language is getting us into trouble.

My "all data in Cybox is unicode" should have been qualified as string data, as clearly int or floats cannot be Unicode.  This "implied" string came from the context we were in, as we were talking about MATCHES, which can not apply to floats or ints, or many other data types.

To my mind, the implications of a "binary regex" are that the regex, which is required to be a string literal, must be convertible to binary as defined in section 4.2.4 (no code points > 255). Higher numbered code points could never match to a byte, so it wouldn't make sense.

This is correct, and how it should be.  From my reading of the PCRE spec, this is how things are handled in the case that a unicode flag is not specified.

Also, many regex features require textual semantics. When applying to sequences of bytes, they don't make sense. E.g. what would the regex \d (match a numeric digit) mean when applied to binary data? You'd have to ascribe textual semantics to various byte values, e.g. use ASCII interpretation. So \d would match byte values in the range 48-57, inclusive.

This is already defined by PCRE (http://www.pcre.org/original/doc/html/pcrepattern.html):
By default, characters whose code points are greater than 127 never match \d, \s, or \w, and always match \D, \S, and \W, although this may vary for characters in the range 128-255 when locale-specific matching is happening. These escape sequences retain their original meanings from before Unicode support was available, mainly for efficiency reasons. If PCRE is compiled with Unicode property support, and the PCRE_UCP option is set, the behaviour is changed so that Unicode properties are used to determine character types, as follows:
So, as you can see, it is well defined by PCRE.

So there are deeper considerations implied by using regexes to match binary data, which aren't spelled out in the spec

Can you please let me know what they are?  I believe I have addressed all of them above.

I was being conceptually pretty pedantic: All of PCRE's documentation seems to be about text.  Your quoted paragraph mentions "characters" and "code points".  How can any of that be applicable when we are not dealing with characters or code points (in binary observable data)?  Binary data is just that: bytes.  There are no characters.  So in that sense, there is no definition in PCRE.  We're off the reservation here, so to speak.
I know programmers blur text and binary types together because it's common in programming languages: C and C++ use "char" (or char-based) type(s) for binary data; and Python2 uses "str".  So it's no wonder it gets so confusing.  But I like keeping them distinct, and cybox does this.  If it were me, I'd go to pains to define separate operations, with separate semantics, and explicit mappings where any kind of conversion had to take place.
So here is more pedantic regex/binary specification (at least, as far as I could solidify it in my brain tonight):

Regexes are inherently textual; many features don't make any sense on binary data.  E.g. \d, \s, etc.  Therefore, binary data must be treated as text for the purposes of pattern matching.  This is done as follows: map byte values 0-127 to characters according to ASCII (or unicode, same thing?), with their usual meanings, and map byte values 128-255 to unspecified characters which have no additional semantics.  This implies that regexes must not rely on any unicode property or feature of any character with codepoint > 127.

Are you allowed to say in the spec that "behavior is indeterminate when locale-specific matching is happening?  Is this what you'd call 'well defined'?  Perhaps this is just a case of not letting perfect be the enemy of the good.

Regexes are expressed as string literals; if a regex string in a pattern has any character with code point > 255, the match must eval to False, per 4.2.4.

(Although, the high-codepoint char could be optional in the pattern, and it would technically not be a problem; it would just never match anything.)

Regexes may attempt to match high codepoint characters in other ways, e.g. \x{...}.  Regexes like this are exempt from the 4.2.4 restriction, but that might never match anything in the target binary sequence.  Evaluation to True is possible, if that character was optional.
Regexes may include checks of Unicode properties....

Not sure what to do about this.  The paragraph you quoted is only part of the story.  E.g. it defines what \d means when unicode property support (PCRE_UCP) is turned on or off, but doesn't define what \p{Nd} means.  I expect it would lose its special meaning without PCRE_UCP, but would it be an error?  Should it be an error?
It may not be possible to disable syntax like this in all cases (e.g. Java's regex API).  If the spec mandates that an error should occur, does that mean such pattern matcher implementations must manually check regexes for \p{...} sequences and produce errors if they are found?

So, I think I've actually reversed course a little bit: the first bullet is actually describing bringing the binary data into the textual realm, instead of trying to push the regex into the binary realm (a "binary regex" as I had initially described it).  I decided that \s, \d, etc are just too common, and they require text.  You must have textual semantics to have any meaning for those features.  So there is no choice.  Maybe you could even ditch the 4.2.4 restriction, if regexes don't have to change?  (Not that that's possible at this point.)
Anyway, I'm not going to try for a perfect Python implementation, I'm just going to use re and hopefully that will be good enough :)  (And it does admit its weakness with regards to unicode, at least in the Python3 re docs.)

Are you allowed to say in the spec that "behavior is indeterminate when locale-specific matching is happening? Is this what you'd call 'well defined'? Perhaps this is just a case of not letting perfect be the enemy of the good.

No, I am not.  I am saying use the Unix standard C locale for matching in this case.

Regexes are expressed as string literals; if a regex string in a pattern has any character with code point > 255, the match must eval to False, per 4.2.4.

Correct.
Please make sure you read: http://www.pcre.org/original/doc/html/pcreunicode.html
There are useful things like:

In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF support, and, in addition, you must call pcre_compile() with the PCRE_UTF8 option flag, or the pattern must start with the sequence (*UTF8) or (*UTF). When either of these is the case, both the pattern and any subject strings that are matched against it are treated as UTF-8 strings instead of strings of individual 1-byte characters.

I was attempting to do some testing on Python3 w/ pcre, but due to brokenness w/ pip, I am unable to verify that passing in a bytes to pcre for matching will work properly.  It should as it's clearly supported when a Unicode flag is not specified per the description mentioned above.
P.S. Using re will be an issue, and if you do, you need to make sure that it is very clear that the code is NOT a reference implementation and has this limitation that regular expressions are not compatible w/ the STIX patterning.

I didn't know there were python bindings for PCRE.  I tried installing python-pcre, but it says you need msvc-14 to build the C part.  Ah well.
As far as binary matching, I think re does the right thing, ignoring unicode properties on codepoints > 127 but using them when matching textual strings.  I think the problems may arise for the more usual textual string matching, e.g. because it doesn't support some features like \p{...} at all.  Perhaps regex would be preferable?
I did some experimentation with Java (just to pick some other commonly used regex package), and was unable to get the correct behavior with the standard java.util.regex package.  Applying regexes to binary data seems like an unusual thing to do for most people, although cybersecurity folk probably do it extensively in looking for their viral signatures.  Makes me wonder what they use (I've never built anything like that).  Probably a lot of custom engines.  Anyway, some people may have to look outside their customary modules to find something that satisfies spec requirements.

Yes, it is necessary that the patterning language supports matching on "binary" data, not just text in order to support many use cases.  There was a large debate about which regex library, and it was decided to use pcre because of the fact that language bindings exist for most libraries, and that most language's regex libraries are not compatible.  We could have spend months defining a new regex language and require everyone to implement a converter to their preferred regex engine, but it was decided that was too complicated, so it was agreed upon to use PCRE as the implementation.

Fixed by #44


Escape menu

made an escape menu, did file management. added a thanks to Limezu in the readme and links to the assets provided by them as well as their itch.io page

Maybe a nit, but I try to never use spaces in file/directory names. e.g. instead of Scenes/escape menu/escape menu.gd, Scenes/escape_menu/escape_menu.gd. Probably not a huge deal here since we're mostly using the Godot editor, and the plz-no-spaces thing makes command line interactions easier.

good point, I'll be sure to look for that in the future. I've updated the file names


Negative PP

I was playing on the 2.0032 beta for a few days and everything was working totally fine until I randomly bought an extra 10 cursors and suddenly half my buildings went into the negatives with their PP, instantly putting the "most negative" up at the top of the list constantly and making the calculations completely wrong.

Buying in tens fixes this, but if you buy ten in one increments it doesn't fix itself which leads me to believe it's a bug rather than some production weirdness going on.

if I buy a group of 10 all that happens is it temporarily fixes itself for that building and just jumps to another building with negative PP
Buying upgrades sometimes fixes it, but most of the time it just changes other buildings to negative in it's place.

Fiddling with average/current cps calculation settings does nothing, doesn't change PP in the slightest.

Sugar lumps seem to not be supported, so when calculating efficiency the new cps after a purchase doesn't include sugar lump effects. If the purchase has less of an impact than sugar lumps it sees the new cps as lower than the current one, resulting in negative pp.  Is this plugin still maintained?

Sorry, on vacation right now.  I'll take a look when I get back.  It is still beta, right?

It is still beta. Supporting only sugar lumps should be an easy fix, but I'm not sure what else is added in beta that can break the plug in.

just as an update, this issue hit the main release today in 2.0034.  I've never contributed to CM before so don't count on it, but I'll take a look to see if I can whip up a quick fix for sugar lumps at least.

I can pretty much confirm that this issue is caused by sugar lumps. They improve CpS by leveling up towers which then confuses CM. Haven't found the minigames and such yet, so I have no idea if there's more things that need fixing. To confirm this specific issue I just leveled up my Prisms once and got the behaviour shown above.

done with a quick fix that just accounts for sugar lump bonuses at https://cdn.rawgit.com/vogon/CookieMonster/776909c31902c30ddd5cae0732645ba51ba06b6a/CookieMonster.js.  (https://github.com/vogon/CookieMonster/tree/v2_0034 is the fork/branch if you want to make sure I didn't sneak anything evil in there.)

Okay, thanks @vogon.  I'm back and I'll take a look at the CC changes and your fixes.

Fixed in Version 2.0034.1

Thanks a lot o/


[FEATURE] Implement Alerts and Notifications from Rancher for Longhorn

Is your feature request related to a problem? Please describe.
This feature request relates to: https://github.com/longhorn/longhorn/issues/928https://github.com/longhorn/longhorn/issues/928
It would be nice to setup Alerts and Notifications in Rancher to get notified about certain things in Longhorn.
Describe the solution you'd like
To be able to create Alerts from within Rancher and attached Notifiers to those Alerts. This would require an integration with Rancher.
Describe alternatives you've considered
Being able to expose and export logs from Longhorn to a central logging tool. This would also be able to trigger alerts, if that tool support such a feature.

@Sheldonwl  I think the user can already do this in v1.1.0, as a part of https://github.com/longhorn/longhorn/issues/1180 , e.g. https://longhorn.io/docs/1.1.0/monitoring/integrating-with-rancher-monitoring/ Can you confirm it's done?

@yasker what do you think about having ServiceMonitor object support in Longhorn's chart? I can create a PR to enable that if you think this is a good idea.

@amnk Unfortunately I am not super familiar with this part. @PhanLe1010 what do you think?

@amnk
Do you mean deploying this service monitoring CR with Longhorn by default? https://longhorn.io/docs/1.1.2/monitoring/integrating-with-rancher-monitoring/#add-longhorn-metrics-to-the-rancher-monitoring-system
It would mean that Longhorn also must be shipped with ServiceMonitor CRD depending on where user cluster already has it or not. IMO, it is best to just let user deploy the service monitor CR by themself if users want to enable monitoring. Similar to what we did with CSI snapshot.

@PhanLe1010 lately a lot of charts include Servicemonitor object in the chart. For example nginx-ingress does that, as well as metallb, coredns, VictoriaMetrics and a lot more.
Of course user can deploy that himself, but including that in the default chart just makes user experience better (imho)

@PhanLe1010 any changes here? It is a great idea to configure a ServiceMonitor in the Helm chart as a lot of other Helm charts do. Of course it can be disabled by default, But it allows users by using only one option enable it. As well as enabling ability to define Prometheus rules is also a great possibility to simplify overall configuration process.

Thanks @amnk and @ybykov-a9s  for the feedback.
We will discuss this proposal with the team
cc @joshimoo @shuo-wu @innobead. Do you have any opinion on this? I agree with users that we should include the service monitor and example Prometheus rules in the Helm chart but disable it by default.

Thanks @amnk and @ybykov-a9s  for the feedback. We will discuss this proposal with the team
cc @joshimoo @shuo-wu @innobead. Do you have any opinion on this? I agree with users that we should include the service monitor and example Prometheus rules in the Helm chart but disable it by default.

+1

@PhanLe1010 I agree if we can make the deployment toggle + user configurable during helm / rancher chart installation and it does not impact upgrade process, it would be a nice to have to expose + ship it.
We can then use the same approach to ship additional optional stuff during installation like network policy, pprof scraping/monitoring, etc down the line.
cc @innobead what do you think?

100% agree.

connect with https://github.com/harvester/harvester/issues/2517
Harvester is planning to add logging and alert for all related componenets.

any news here?


Didn't work when supplying a wifi connection string - zsh killed the session due to timeout

Tried using the following:
generator -content "WIFI:S:mYwIfI;T:WPA;P:secret_passwordt;H:false;;" -svg-optimized /Users/me/support/wifi.svg
Worked when using a simple string.

The wifi string worked when NOT adding -svg-optimized /Users/me/support/wifi.svg

Just using -svg works as well. -svg-optimized does not work at all.

That works - thanks!