Remove kzg point and blob validations in gossip

The c-kzg library has moved to all bytes interface after https://github.com/ethereum/c-kzg-4844/pull/62 which also removes the bytes_to_g1 and bytes_to_bls_field from the public interface.
My understanding is that the conversion function was removed from the public interface since verify_aggregate_kzg_proof will contain the checks for if the commitments and proofs are valid g1 points and each blob < BLS_MODULUS so we do not need to make a separate check.

Yes, this makes sense. One question is if we should address this:

The KZG commitments of the blobs are all correctly encoded compressed BLS G1 points

Without this check, for any blobs out of the data retention window, there will be no check whether they are valid KZG commitments. We could add a function to check this in the BLS library.
As far as I can see this adds no additional security benefits but wanted to raise it.

Without this check, for any blobs out of the data retention window, there will be no check whether they are valid KZG commitments.

my assumption was that we would not gossip any blobs outside of the availability period, and if so, this would not be a concern as a gossip validation

Without this check, for any blobs out of the data retention window, there will be no check whether they are valid KZG commitments.

my assumption was that we would not gossip any blobs outside of the availability period, and if so, this would not be a concern as a gossip validation

Blocks are gossiped, and their KZG commitments will not be verified to be G1 points outside this window. As I mentioned, I do not think this has any security impact so I think it's fine to leave it.

Blocks are gossiped

blocks aren't gossiped outside of a 32 slot window which is << the pruning window.
so all good


added bolts framework

https://github.com/BoltsFramework/Bolts-iOS

Pretty vague, @clsource can you comment on how bolts is useful

@dkhamsing  I just copied and pasted some excerpt of the docs xd.
Bolts includes:

"Tasks", which make organization of complex asynchronous code more manageable. A task is kind of like a JavaScript Promise, but available for iOS and Android.

An implementation of the App Links protocol, helping you link to content in other apps and handle incoming deep-links.

Thanks


Can't get store pemfiles working

Issue description
I've just upgraded from v<IP_ADDRESS> to v<IP_ADDRESS>4 and I need the additional pem files for use in hMailServer on Windows Server 2016.
However, even though I specify the --store pemfiles switch as described on https://github.com/PKISharp/win-acme/wiki/Store-plugins no files are stored in the output folder except the 2 default files.
Steps to reproduce
I've tried to run wacs.exe with various switches (with or without pemfilespath) and also updating the settings.config file to try different paths.
Example:
wacs.exe --store pemfiles --pemfilespath c:\ProgramData\Certificates\ --force --verbose
Output:
[DBUG] Scanning IIS sites
[DBUG] Send POST request to https://acme-v02.api.letsencrypt.org/acme/new-order
[DBUG] Send GET request to https://acme-v02.api.letsencrypt.org/acme/authz/PgKF0TJ2itxEB2px38ZZBMX0xcqu7TkdiyYUQMpJzIU
[INFO] Authorize identifier: tools.mydomain.dk
[INFO] Cached authorization result: valid
[DBUG] Certificate store: WebHosting
[DBUG] RSAKeyBits: 3072
[DBUG] Send POST request to https://acme-v02.api.letsencrypt.org/acme/finalize/57539919/463745753
[INFO] Requesting certificate [IISSite] tools.mydomain.dk
[INFO] Installing certificate in the certificate store
[DBUG] Opened certificate store WebHosting
[INFO] Adding certificate [IISSite] tools.mydomain.dk 2019-5-21 21:20:57 to store WebHosting
[VERB] CN=tools.mydomain.dk - CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US (80E6C5053B2B08E919C338D7C9853050ED5AB90F)
[VERB] CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US - CN=DST Root CA X3, O=Digital Signature Trust Co. (E6A3B45B062D509B3382282D196EFE97D5956CCB) to CA store
[VERB] CN=DST Root CA X3, O=Digital Signature Trust Co. - CN=DST Root CA X3, O=Digital Signature Trust Co. (DAC9024F54D8F6DF94935FB1732638CA6AD77C13) to AuthRoot store
[DBUG] Closing certificate stores
[DBUG] Scanning IIS sites
[INFO] Installing with IIS...
[INFO] Updating existing https binding tools.mydomain.dk:443
[INFO] Committing 1 https binding changes to IIS
[INFO] Next renewal scheduled at 2019-7-15 21:20:57
[INFO] Renewal for [IISSite] tools.mydomain.dk succeeded
settings.config file
<PKISharp.WACS.Properties.Settings>   <setting name="FileDateFormat" serializeAs="String">     <value>yyyy/M/d H:mm:ss</value>   </setting>   <setting name="RSAKeyBits" serializeAs="String">     <value>3072</value>   </setting>   <setting name="PageSize" serializeAs="String">     <value>50</value>   </setting>   <setting name="CertificatePath" serializeAs="String">     <value>C:\ProgramData\Certificates\</value>   </setting>   <setting name="CleanupFolders" serializeAs="String">     <value>True</value>   </setting>   <setting name="PrivateKeyExportable" serializeAs="String">     <value>True</value>   </setting>   <setting name="ConfigurationPath" serializeAs="String">     <value />   </setting>   <setting name="Proxy" serializeAs="String">     <value>[System]</value>   </setting>   <setting name="RenewalDays" serializeAs="String">     <value>55</value>   </setting>   <setting name="DefaultCertificateStore" serializeAs="String">     <value />   </setting>   <setting name="ScheduledTaskExecutionTimeLimit" serializeAs="String">     <value>02:00:00</value>   </setting>   <setting name="ScheduledTaskRandomDelay" serializeAs="String">     <value>00:00:00</value>   </setting>   <setting name="ScheduledTaskStartBoundary" serializeAs="String">     <value>01:58:00</value>   </setting>   <setting name="ClientName" serializeAs="String">     <value>win-acme</value>   </setting>   <setting name="ProxyUsername" serializeAs="String">     <value />   </setting>   <setting name="ProxyPassword" serializeAs="String">     <value />   </setting>   <setting name="EncryptConfig" serializeAs="String">     <value>True</value>   </setting>   <setting name="DefaultBaseUri" serializeAs="String">     <value>https://acme-v02.api.letsencrypt.org/</value>   </setting>   <setting name="DefaultBaseUriTest" serializeAs="String">     <value>https://acme-staging-v02.api.letsencrypt.org/</value>   </setting>   <setting name="DefaultBaseUriImport" serializeAs="String">     <value>https://acme-v01.api.letsencrypt.org/</value>   </setting>   <setting name="HostsPerPage" serializeAs="String">     <value>50</value>   </setting>   <setting name="ECCurve" serializeAs="String">     <value>secp384r1</value>   </setting>   <setting name="SmtpServer" serializeAs="String">     <value>localhost</value>   </setting>   <setting name="SmtpPort" serializeAs="String">     <value>25</value>   </setting>   <setting name="SmtpUser" serializeAs="String">     <value />   </setting>   <setting name="SmtpPassword" serializeAs="String">     <value />   </setting>   <setting name="SmtpSecure" serializeAs="String">     <value>False</value>   </setting>   <setting name="SmtpSenderName" serializeAs="String">     <value>Win-Acme</value>   </setting>   <setting name="SmtpSenderAddress" serializeAs="String"><EMAIL_ADDRESS> </setting>   <setting name="SmtpReceiverAddress" serializeAs="String"><EMAIL_ADDRESS> </setting>   <setting name="DefaultCentralSslStore" serializeAs="String">     <value />   </setting>   <setting name="DefaultCentralSslPfxPassword" serializeAs="String">     <value />   </setting>   <setting name="DefaultPemFilesPath" serializeAs="String">     <value>C:\ProgramData\Certificates\</value>   </setting>   <setting name="EmailOnSuccess" serializeAs="String">     <value>True</value>   </setting>   <setting name="DeleteStaleCacheFiles" serializeAs="String">     <value>False</value>   </setting>   <setting name="DnsServer" serializeAs="String">     <value />   </setting> </PKISharp.WACS.Properties.Settings>
Only 2 files are written to the folder: xyz-cache.pfx and xyz-csr.pem. I need the chain and key pem files as well.
What could be the problem and how can I resolve it?
Thanks in advance.

I think this stems from a misconception about how plugins work. The idea is that you pick and configure them while setting up the renewal. From that point forwards those choices are "locked in" for that particular certificate.
You can modify/overwrite the renewal if you want to alter the parameters, but you cannot renew and change them at the same time.
So if you want to use PemFiles instead of the CertificateStore, you have to re-create the renewal with the correct parameters/choices.
Note though that you cannot use two stores at the same time and that IIS cannot work with PemFiles. If you want to have both, you will have to set up two separate renewals for the same set of domains.

Note though that you cannot use two stores at the same time and that IIS cannot work with PemFiles. If you want to have both, you will have to set up two separate renewals for the same set of domains.

This sounds like a step back compared to v1.9.x. I also need the pem files for other software next to IIS. Previously all files were available in the configuration path but now only cryptically named pfx and csr are stored there.

I see your point, but the 1.9.x way was not perfect for several reasons.

There would be no less than eight files per renewal in a folder which was very ambiguous: some were for internal use, some were useful for some users, and some didn't seem to have any function at all. On large deployments this led to file system issues because simply enumerating the folder would become too slow.
While the names were more readable, there were at risk of inconsistencies (being out of date) and collisions (two renewals with the same name) that couldn't be properly fixed.
Storing private keys in an unprotected form in a directory readable by all users was a horrible default configuration from a security perspective.

The new approach makes thinks very clear:

The cache folder is for internal use.
If you want your certificate to be somewhere, you have to be explicit about where you want it and in which format.

I can imagine that there would be scenarios where you'd need the exact same certificate in both .pem and .pfx formats, in which case support for multiple store steps would need to be added (like you can pick multiple install steps). But for most cases setting up a second renewal should be a reasonable alternative. They're free after all 😉

I agree with Marco. It seems like a step back featurewise and the solution adds unnecessary complexity from a user perspective and is somewhat a waste of resources - even though they're "free". I can also see the common sense in the points Wouter makes, though, so it's a bit tricky.
Could it be a feature request to restore that specific behavior from the v1.9 branch via a new command line switch or something?
I've attempted to do as suggested to get a second certificate with pem files, but I can't seem to get it working correctly. I'm trying the following command - if I've understood it:
wacs.exe --target manual --validation filesystem --host sub.domain.dk --store pemfiles --pemfilespath C:\ProgramData\Certificates\ --webroot D:\inetpub\wwwroot\sub.domain.dk --verbose
The result is this error:
 [EROR] Preliminary validation failed System.Net.WebException: The remote server returned an error: (404) Not Found.    at System.Net.HttpWebRequest.GetResponse()    at PKISharp.WACS.Plugins.ValidationPlugins.HttpValidation2.GetContent(Uri uri)
at PKISharp.WACS.Plugins.ValidationPlugins.HttpValidation2.WarmupSite()    at PKISharp.WACS.Plugins.ValidationPlugins.HttpValidation2.PrepareChallenge()`
A folder named ".well-known" is created in the webroot path, but it seems to remain empty at all times.
Sorry for my ignorance if I've misunderstood something.
Thanks in advance.

@WouterTinus I agree that it's not a good default to store these files unprotected in the filesystem. Unfortunately most "other" software like mail/ftp-servers need the files like this. Most software does not even support a password protected private key.
So writing .pem files should not be default but it should not be too hard for people that need it. Setting up a second renewal is of course possible but sounds a bit like a hack / workaround. How about an option to also export the certificate in the format the user likes during renewal?  So next to the store that is used for storage and renewal, it can also write the files in any other format but it won't use that as "the store".
Or maybe an option to export an existing certificate from the store to .pem files in a specific location?  (or did I miss that and is that already possible?)

Or maybe an option to export an existing certificate from the store to .pem files in a specific location? (or did I miss that and is that already possible?)

Let's Encrypt all is about automation, so if it cannot be done automatically then the value of such a feature would be very limited.

How about an option to also export the certificate in the format the user likes during renewal? So next to the store that is used for storage and renewal, it can also write the files in any other format but it won't use that as "the store".

That's a bigger hack than separate renewals. Many people are already struggling to understand the concept of the different steps in the renewal process, so I believe that splitting store into an 'A' and a 'B' type with different purposes would make things worse.
The point of the pipeline of plugins is to enable users to mix and match. E.g. if someone wants to write an Apache installer, they should be able to that without making a 'B' installer that works with the 'B' store.
The proper solution is to support multiple, fully equivalent stores for a renewal, so to make all selected stores usable by all selected installation plugins. Anything less is just not future-proof.

The above changes have been released in 2.0.7, so you will now be able to pick multiple stores for a single renewal.


Unify how the k8s package files work

Related to https://github.com/celestiaorg/knuu/labels/refactor

would you please share more context on this?

would you please share more context on this?

It's basically what you already did the function names being more uniform.


Serializing and deserializing a bot instance

🚀 The feature
Can you add the possibility to serialise and deserialise bots instances. This may be useful in web development context.
Motivation, pitch
One of the bottlenecks we face is the current implementation of bot instances, which are not easily serializable. This makes it difficult to store the bot's state, share it across multiple server instances, or even keep a reliable session for the user, thus affecting both performance and user satisfaction.
Benefits:
Scalability: Being able to serialize and deserialize a bot instance will allow us to more efficiently use our server resources. This lays the foundation for load balancing and ensures that as our user base grows, our services can easily scale to meet the demand without performance hitches.
State Preservation: A serializable bot instance means that user interaction can be picked up right where it left off, providing a much more fluid and engaging user experience.
Error Recovery: In the event of a system failure, a serialized instance can be quickly restored, thereby reducing downtime and improving overall system reliability.
Data Insights: Serialized data can be more easily analyzed to gain insights into user behavior and bot performance, providing valuable data that can be used to improve services.
Reduced Latency: Quick serialization and deserialization mean faster response times for the end-user, thereby enhancing the overall user experience.
Technical Outline:
Serialization: The feature should provide a straightforward method to convert a bot instance into a serializable format, e.g., JSON, that captures all the necessary state information.
Deserialization: It should also offer an equally straightforward method to restore a bot instance from its serialized format back to a fully functional object in memory.
Compatibility: The feature should be compatible with our existing systems, requiring minimal changes to integrate.
Security: Serialized data should be encrypted to ensure that sensitive user data is fully protected.

I will pick this up.

any suggestions for the format? anything I should look at for inspiration?
since you speak of error recovery, do you want the bot to store a serialized state after every request?
do you want to serialize the database contents as well? I'd guess database backups are a different topic.
security: how do you think this should work? I assume that the bot is accessed through an API, so the serialized dumps are not exposed. Let's say it's encrypted using a secret key. If you store that key in your backend machine, it will be stored with the serialized data, leaving your with no security gain, if you don't store it, you can't automatically serialize and deserialize. Just some of my intial thoughts, I'm sure there's an answer to that.

Format
JSON: Given that I am working in a web development context and considering the widespread compatibility and human-readability, JSON would be a convenient choice for serializing bot instances.
Error Recovery
I think we should store serialized states at periodic intervals rather than after every request to balance between resource consumption and error recovery capabilities (make periodic snapshots).
You are right, we should probably just focus on these two aspect, database backups and security are a different topic, indeed.


Crash on game over

The game crashed when the AI shot the player's light. A vector operation went out of range. Have not been able to reproduce.

To be more precise, the game crashes when there are several bots  in the game and the player is being attacked and killed while being shot at by several bots.


Python: Resolve hanging issue in Jupyter notebooks when re-running async functions

Motivation and Context
Resolves #4137. The issue was that if you try to re-run the sample joke in the 00-getting-started notebook, the subsequent call to the joke hangs.

Description
There were issues when re-running asynchronous functions (_invoke_semantic_async and _invoke_native_async) due to improper handling of asyncio event loops in a multi-threaded env. Specifically, the code had an issue when an event loop was already running, leading to hanging or blocking behavior on subsequent runs.
The change replaced the old code Python's ThreadPoolExecutor, which simplifies thread management. It handles the creation, execution, and cleanup of threads more efficiently and safely.
The new run_async_in_executor method ensures that each async function runs in its own new event loop, avoiding conflicts with the main thread's event loop or other threads.

Contribution Checklist

[X] The code builds clean without any errors or warnings
[X] The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
[X] All unit tests pass, and I have added new tests where possible
[X] I didn't break anyone :smile:

I think this fix has caused a new problem. When trying to run the semantic functions with input parameters from this notebook [https://github.com/microsoft/semantic-kernel/blob/main/python/README.md], my event loop hangs now:
In addition, when trying to run this code as a standalone python file, I get RuntimeError: There is no current event loop in thread 'MainThread'.
I'm trying to find the exact problem and fix associated with this but reporting here so the community can also look into it.

Hi, @BeeShall. Are you referring to this code?. Can you send us the exact code you're trying to run as a standalone python file? Thanks.

I think this fix has caused a new problem. When trying to run the semantic functions with input parameters from this notebook [https://github.com/microsoft/semantic-kernel/blob/main/python/README.md], my event loop hangs now:
In addition, when trying to run this code as a standalone python file, I get RuntimeError: There is no current event loop in thread 'MainThread'.
I'm trying to find the exact problem and fix associated with this but reporting here so the community can also look into it.

I am seeing the same issue as @BeeShall .
And yes, @moonbox3 , the code is very similar to that example. Let me write the code and I'll send it over

`import semantic_kernel as sk
from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
kernel = sk.Kernel()
deployment = "gpt35turbo"
api_key = ""
endpoint = ""
azure_text_service = AzureChatCompletion(deployment_name=deployment, endpoint=endpoint, api_key=api_key)    # set the deployment name to the value of your text model
kernel.add_chat_service("dv", azure_text_service)
prompt = """{{$input}}\n\nEXPLAIN THESE SYSTEM LOGS IN A HUMAN READABLE FORMAT."""
prompt = "{{$input}}/n/nSummarize this"
log_explainer = kernel.create_semantic_function(prompt_template=prompt, max_tokens=2000, temperature=0.2, top_p=0.5)
for i in range(10):
print(log_explainer(f"ERROR: {i}"))`

@moonbox3 Yes that's the one. keeps hanging in Jupyter notebook in a thread lock situation and I get that error when I try to run standalone.
@NatanMish yep that's exactly how I see it too.

@BeeShall, thanks for confirming. And @NatanMish, thank you for the code. Appreciate you pointing this out. This is fixed in #4485.


Fixed Typo: Feach → Fetch

A simple typo fix.

@don4of4 : Thanks for your contribution! The author, @linda33wj, has been notified to review your proposed change.

@don4of4 - thanks for catching that!


22.11

PR to prepare everything for 22.11.

The state version is a sensible value that should explicitly set in the users config. Following home-manager: https://github.com/nix-community/home-manager/pull/3043.
Updating qemu, nix and pinned nixpkgs. Store path to proot updated because not building, see below
Updating references of 22.05 to 22.11
Add stateVersion 23.05 (because master after release-22.11 will essentially be 23.05 (also following home-manager's approach)

Currently, bootstrap is not building:
$> nix build .#bootstrapZip --impure -L
nix-directory> unpacking sources
nix-directory> unpacking source archive /nix/store/4hk7sv1pdp8ip0ssnk88ri91fd2d9k4p-nix-2.11.1-aarch64-linux.tar.xz
nix-directory> source root is nix-2.11.1-aarch64-linux
nix-directory> setting SOURCE_DATE_EPOCH to timestamp<PHONE_NUMBER> of file nix-2.11.1-aarch64-linux/install-systemd-multi-user.sh
nix-directory> patching sources
nix-directory> configuring
nix-directory> no configure script, doing nothing
nix-directory> building
nix-directory> terminate called after throwing an instance of 'nix::SysError'
nix-directory>   what():  error: reading symbolic link '/proc/self/exe': No such file or directory
error: builder for '/nix/store/5qnqr259jbqsq95lfwc7ln0nh5lb407i-nix-directory.drv' failed with exit code 255;

Issue seems to be on nix-store --init after nixpkgs update, see https://github.com/t184256/nix-on-droid/blob/master/pkgs/nix-directory.nix#L46

Found a solution:

mounting /proc helps, not sure about other side effects
needed to create the nix state directory (mkdir --parents ${buildRootDirectory}/nix/var/nix/db) in order to tell nix to use the manually created /nix directory instead of a chroot nix store in /homeless-shelter/.local/share/nix/root. related warning that is printed without this fix:warning: '/nix' does not exist, so Nix will use '/homeless-shelter/.local/share/nix/root' as a chroot store

BTW there are quite some old branches that are merged and/or maybe not needed anymore. Except for current open PR branches, your testing branch and the release-* branches (and maybe pre-module-system branch), are the other branches still required?
Just for some house-keeping :)
  remotes/origin/FAILED-on-device-proot-building a8c3881 Attempt to fix proot static compilation
  remotes/origin/deploy-fix                      1194ecf deploy: allow specifying urls not ending with source.tar.gz
  remotes/origin/does-this-fix-make              d627071 pkgs/cross-compiling/proot: fix seccomp+setresxid
  remotes/origin/fix-flake-prompt                b623074 login-inner: fix default choice on bootstrap flakes prompt
  remotes/origin/fix-profile                     048b22f environment.path: fix installation with nix profile
  remotes/origin/master                          4e58ce7 fix(terminal.nix): quote font paths
  remotes/origin/pre-module-system               a505862 Add less to basic-environment, nix needs a $PAGER.
  remotes/origin/prerelease-20.09                fed9bbe pkgs/qemu-aarch64-static: update
  remotes/origin/prerelease-22.05                b286ac1 pkgs/nix-directory: update nix
  remotes/origin/prerelease-22.05-pt2            0ad6321 pkgs: use non-prebuilt SDK
  remotes/origin/prerelease-22.05-unmerged-work  d0b08d2 proot-termux: simplify module
  remotes/origin/prerelease-22.11                8816289 bootstrap: update nixpkgs version
  remotes/origin/release-19.09                   d551f63 Add a comment to clarify setting user.shell
  remotes/origin/release-20.03                   b651458 Fix Cachix in Travis with `trusted-users = travis`
  remotes/origin/release-20.09                   0dec5d4 modules/user: use runCommandLocal for idDerivation
  remotes/origin/release-21.05                   3e3a4e2 overlays/lib/nixpkgs.nix: use system
  remotes/origin/release-21.11                   e87f42e modules/upgrade: fix missing /dev/shm on upgrade from <21.11
  remotes/origin/release-22.05                   d49fd3a CHANGELOG: Release 22.05
  remotes/origin/testing                         62cf934 fixup ssh: don't write to stdout
  remotes/origin/update-sdk                      dda3bd3 WIP: compile proot dynamically with non-prebuilt SDK
  remotes/origin/wip-non-static-proot            07a89d7 WIP non-static proot

I would also recommend to use more variables for the nixos version for the next update.

What do you mean by that? What kind of variables?

BTW there are quite some old branches that are merged and/or maybe not needed anymore.

Didn't know I had so many failed attempts to update SDK and build proot differently =/ Cleaned up some of the branches.

Rebased and added a commit for #230. Also dropped the workaround for default nix version < 2.4. If that is fine, you can merge and do the usual release steps :)


CB-14181: (android) Fix bug - Cannot read property 'filesystemName' of null

fix error :
Error in Success callbackId: File1539060614 : TypeError: Cannot read property 'filesystemName' of null
cordova.js:314 Uncaught TypeError: Cannot read property 'filesystemName' of null
The fsRoot might be null. reproduced with latest cordova on android emulator.

Platforms affected
What does this PR do?
What testing has been done on this change?
Checklist

[x] Reported an issue in the JIRA database
[x] Commit message follows the format: "CB-3232: (android) Fix bug with resolving file paths", where CB-xxxx is the JIRA ID & "android" is the platform affected.
https://issues.apache.org/jira/browse/CB-14181
[ ] Added automated test coverage as appropriate for this change.

There are eslint issues with this patch, see Travis CI build logs. Run npm test you will see them.

fixed, yet I see tests are failing and they fail in all other PR's as well.

This will prevent the error, but what will cause the "Cannot read property 'filesystemName' of null"? How can this be reproduced?

@jcesarmobile it has been reproduced to me constantly on android emulator nexus 5
device  - nexus 5
android 7.1.1 nougat (yet will be reproduced on oreo as well I'm sure)
cordova 7.1.0
with the following code from the blog - documentation  (caused on the line fs.root.getFile
        window.requestFileSystem(window.LocalFileSystem.TEMPORARY, 0, (fs) => {
            console.log('file system open: ' + fs.name);
            fs.root.getFile(
                `${window.cordova.file.applicationStorageDirectory}${this.fileName}`, { create: true, exclusive: false }, (fileEntry) => {
                    console.log('fileEntry is file? ' + fileEntry.isFile.toString());
                    var oReq = new XMLHttpRequest();
                    // Make sure you add the domain name to the Content-Security-Policy <meta> element.
                    oReq.open('GET', cvURL, true);
                    // Define how you want the XHR data to come back
                    oReq.responseType = 'blob';
                    oReq.onload = (oEvent) => {
                        var blob = oReq.response; // Note: not oReq.responseText
                        if (blob) {
                        // Create a URL based on the blob, and set an <img> tag's src to it.
                            var url = window.URL.createObjectURL(blob);
                            document.getElementById('bot-img').src = url;
                            // Or read the data with a FileReader
                            var reader = new FileReader();
                            reader.addEventListener('loadend', () => {
                            // reader.result contains the contents of blob as text
                            });
                            reader.readAsText(blob);
                        } else console.error('we didnt get an XHR response!');
                    };
                    oReq.send(null);
                }, function (err) { console.error('error getting file! ' + err); });
        }, function (err) { console.error('error getting persistent fs! ' + err); });

I couldn't reproduce with that code on real devices or the emulator.
Anyway, the code is ok, going to merge.
But what should worry us is why root is null on first place.
As you can reproduce, would be good if you can investigate and create a new issue with more information about what might cause the root to be null.


Allow to filter PR  job results by type

Description
in some projects there are a lot of github actions running and finding those which errored requires scrolling thru a huge list.

would be great if I could somehow filter the list by result (succeeded, errorred, skiped) to only show me the errored ones
Screenshot
No response
Example URLs
https://github.com/phpstan/phpstan-src/pull/1558

We kinda have an open issue for this: #3136

Duplicate of https://github.com/refined-github/refined-github/issues/5621


Reference to 1.7.4-alpha-37 is incorrect

When trying to build this project I get an error which says that version "1.7.4-alpha-37" could not be found. The version available on NuGet under 'prerelease' is "1.7.4-pre-jun-28-2015"

The 1.7.4-alpha pre-release packages are from netofficefw-dev feed. This feed is set in NuGet.config file so NuGet should be able to find it.
Which sample is not working?

I see, the Excel samples are using 1.7.4-alpha package and the exact version was deleted by retention policies. I will update samples.

It should work now (commit baa093c851a581f4c0e207d02cb5dca0328b6ccc)

All good. Thank you for the quick fix and a great project.


Bump PHP requirement in plugin

Task Description

Need to bump the PHP 5.6 requirement in two places:
https://github.com/google/web-stories-wp/blob/c4b2f02b98713b17962f283a5f2205f93d56fbce/web-stories.php#L17
https://github.com/google/web-stories-wp/blob/c4b2f02b98713b17962f283a5f2205f93d56fbce/web-stories.php#L50

There is also
Remove from unit tests.
https://github.com/google/web-stories-wp/blob/c4b2f02b98713b17962f283a5f2205f93d56fbce/.github/workflows/tests-unit-php.yml#L48
https://github.com/google/web-stories-wp/blob/c4b2f02b98713b17962f283a5f2205f93d56fbce/.github/workflows/tests-unit-php.yml#L56
Remove it from composer
https://github.com/google/web-stories-wp/blob/c4b2f02b98713b17962f283a5f2205f93d56fbce/composer.json#L43
By changing this value in composer.json, we should also build the composer lock file and see what dependencies we can remove / update.
Update coding standards.
On a similar note we can also change this line
https://github.com/google/web-stories-wp/blob/c4b2f02b98713b17962f283a5f2205f93d56fbce/phpcs.xml.dist#L61-L62
We can then review what languages, like Scalar type declarations, Return type declarations and Group use declarations we would want to use. This is an epic in it's self.

Yup! I split these all up into different tickets, and turned #7551 into an epic.

Remove from unit tests.

#7599

Remove it from composer

#7594

Update coding standards.

#7595

We can then review what languages, like Scalar type declarations, Return type declarations and Group use declarations we would want to use. This is an epic in it's self.

#7597

Note: Do not merge before 1.8.0


Reverting starting of index persistor thread when recovering index segments - PR https://github.com/linkedin/ambry/pull/2623

We have changed to start index persistor thread in background when recovering index segments. But, some of the data structures (the last offset of a log segment) are not correctly populated while recovery. This causes exception as below
LogEndOffset Name = [1215_0] Offset = [18] before flush cannot be less than currentEndOffSet of index Name = [1215_0] Offset = [624188832]
at com.github.ambry.store.PersistentIndex$IndexPersistor. write (PersistentIndex.java:2616)
at com.github.ambry.store.PersistentIndex$IndexPersistor. run (PersistentIndex.java:2665)
at java.util.concurrent.Executors$RunnableAdapter. call (Executors.java:515)
at java.util.concurrent.FutureTask. runAndReset (FutureTask.java:305)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. run (ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor. runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker. run (ThreadPoolExecutor.java:628)
at java.lang.Thread. run (Thread.java:834)

Reverting back the fix for now

Codecov Report
Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (400b0ba) 72.08% compared to head (945b46f) 19.54%.

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #2636       +/-   ##
=============================================
- Coverage     72.08%   19.54%   -52.54%
+ Complexity    11414     2778     -8636
=============================================
  Files           806      806
  Lines         65351    65351
  Branches       7966     7966
=============================================
- Hits          47105    12775    -34330
- Misses        15653    51392    +35739
+ Partials       2593     1184     -1409

Files
Coverage Δ

...n/java/com/github/ambry/store/PersistentIndex.java
0.00% <0.00%> (-90.93%)
:arrow_down:

... and 593 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


test all different pages

https://github.com/threefoldtech/rivine-chain-explorer/issues/12

closed in https://github.com/threefoldtech/rivine-chain-explorer/commit/940a80b7722d98d9db0b7906e67b8f1b76be3d80


Best Practices: Error Responses from Inbound filters

Are there best practices around how to respond from inbound filters. Let's say for example, there is a inbound filter to check if a particular route exists and if not it needs to respond back with a 404. The only way to do this right now seems to be to create a new endpoint filter that'd return a 404 and then set that as the endpoint on the request context?
Isn't there a better way to do this? Like just throw an exception with say a 404 code or something?

You can create a generic static endpoint filter and use the context to pass in body, status, etc. You can route to it similarly to how the sample does proxy endpoint routing -- based on something in the context.

@artgon can we get the response body info from the HttpOutboundSyncFilter HttpResponseMessage response API?

Yes. response.getBody() or response.getBodyAsText().

thanks ，@artgon no matter what content response ，the body is always null，if I have to setBody or setBodyAsText？so how can I get the body content from the original server？

I'm not sure what you're asking. Are you setting the body or getting the body?

I think I am on the same page with @qtwang as I am trying to understand how Zuul 2 works. I am not able to get the body content from the HttpOutBoundSyncFilter using response.getBody() or response.getBodyAsText() after calling to the origin server from the ProxyEndpoint. It is giving Null value. Look like ProxyEndpoint is still waiting to receive the data from the origin server while Zuul continues to handle the rest of the filters. Any explanation on how to get the body content and do the modification?

@artgon  yes，I am asking the same question as @danesavot ask below .

Can you share some code to reproduce the issue? I can't make it happen on the sample app.

Ok, I think I see what's missing here. For both inbound and outbound filters, if you're getting a body and you want to print it all out in the filter, you need to override the needsBodyBuffered() method and return true. See parent class here: https://github.com/Netflix/zuul/blob/2.1/zuul-core/src/main/java/com/netflix/zuul/filters/BaseFilter.java#L130
I'll update the wiki to include this. Thanks for bringing it to my attention.

I have figured out to get the body content using needsBodyBuffered() last week after I dived into the source codes but I totally forget to update the answer! Anyway, thanks for the response! Hope, you will be more responsive next time :)


When npm start, show "Plugin "react" was conflicted between "package.json » eslint-config-react-app..."bug

Plugin "react" was conflicted between "package.json » eslint-config-react-app » D:\d\react\test_react\node_modules\eslint-config-react-app\base.js" and "BaseConfig » D:\d\react\test_react\node_modules\eslint-config-react-app\base.js".

@7889zcm try to update setupTests.ts with this import statement: import @types/jest;

Thanks, but I'm not sure. What exactly do in setupTest.ts？Just import @types/jest ？

Thanks, but I'm not sure. What exactly do in setupTest.ts？Just import @types/jest ？

U are right, I'm in middle of setting up monorepo wi/ lerna and things are not going well., so I was figuring things out Actually u don't need that import, but, u need to set skip eslint flag: DISABLE_ESLINT_PLUGIN=true in your start script.
Do this, in case u have custom .eslintrc file. Also I've remove CRA eslint config inside package.json.

oh yes thanks

remove eslint config .eslintrc.{js,yml,json} file in your project, CRA provides eslint config. If you want to override the base CRA esling config refer to this section of docs https://create-react-app.dev/docs/setting-up-your-editor/#extending-or-replacing-the-default-eslint-config

data not shown in list


Remove "openSUSE-42"

This distro is EOL (Jan 2018) and should no longer be offered up in any capacity.

Any update on this (at least time-frame wise)?
Is something like this waiting on a new version of the wsl binary to be published?
Thanks! :)

@sbradnick - you are guys planning on publishing new versions of your distro?

"openSUSE-42" appears in wsl --list --online, we've had a hard time getting a definitive answer on how those can be manipulated (or how they got there) ; last answer we received was the removal has to be done as part of a general Windows/WSL update (tied to the binary in some way). This is my way of trying to ensure that happens, if this repo is somehow tied to that process. Regardless, NO ONE should be using the 42 series, it's EOL
openSUSE Leap 42.3 - Jul 1st 2019
openSUSE Leap 42.2 - Jan 26th 2018
openSUSE Leap 42.1 - May 17th 2017
We already have current/(current - 1) versions in the Store:

Tumbleweed
Leap 15.4
Leap 15.3
SLES 15SP4
SLES 15SP3
SLES 12SP5

Ok, works for me.


npm run build fails

My initial intention was to contribute by submitting a PR. But, I ran into some issues on some points which are needed to do so:
Version info
Angular: 6.1
Firebase: 3.17.4
AngularFire: 5.0-rc.11
node: 9.5.0
Other (e.g. Ionic/Cordova, Node, browser, operating system): no
My OS: windows 10**
Steps to set up and reproduce

I just forked the repository
then I git cloned it.
then I npm installed it
then, npm run build fails directly.

Debug output
** Errors in the JavaScript console **
jct@CFLX-22 MINGW64 ~/code/16-angularfire2/angularfire2 (master)
$ npm test

<EMAIL_ADDRESS>test C:\Users\jct\code\16-angularfire2\angularfire2
npm run build && karma start --single-run

<EMAIL_ADDRESS>build C:\Users\jct\code\16-angularfire2\angularfire2
rm -rf dist && node tools/build.js

events.js:137
throw er; // Unhandled 'error' event
^
Error: spawn node_modules/.bin/ngc ENOENT
at _errnoException (util.js:1003:13)
at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
at onErrorNT (internal/child_process.js:389:16)
at process._tickCallback (internal/process/next_tick.js:152:19)
at Function.Module.runMain (module.js:703:11)
at startup (bootstrap_node.js:190:16)
at bootstrap_node.js:662:3
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>build: rm -rf dist && node tools/build.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!      C:\Users\jct\AppData\Roaming\npm-cache_logs\2018-07-31T09_58_56_550Z-debug.log
npm ERR! Test failed.  See above for more details.
jct@CFLX-22 MINGW64 ~/code/16-angularfire2/angularfire2 (master)
$ ng version
Angular CLI: 6.0.0
Node: 9.5.0
OS: win32 x64
Angular: 6.1.0
... common, compiler, compiler-cli, core, platform-browser
... platform-browser-dynamic, platform-server
Package                      Version
@angular-devkit/architect    0.6.0 (cli-only)
@angular-devkit/core         0.6.0 (cli-only)
@angular-devkit/schematics   0.6.0 (cli-only)
@schematics/angular          0.6.0 (cli-only)
@schematics/update           0.6.0 (cli-only)
rxjs                         6.2.2
typescript                   2.7.2
jct@CFLX-22 MINGW64 ~/code/16-angularfire2/angularfire2 (master)
$ firebase --version
3.17.4
jct@CFLX-22 MINGW64 ~/code/16-angularfire2/angularfire2 (master)
$ node -version
C:\Program Files\nodejs\node.exe: bad option: -version
jct@CFLX-22 MINGW64 ~/code/16-angularfire2/angularfire2 (master)
$ node -v
v9.5.0
Expected behavior
It should run ngc command in node_modules/.bin/ngc, but does not.
Actual behavior
It fails as the log indicates.
Is there anything that I did wrong ? I just followed what was indicated on the CONTRIBUTING.md page.

remove your node_modules then try to run npm test again

I believe we need to do some work to make our tooling Windows friendly.

yeah, forgot it, I build successful on Mac. it must be Window problem

Thanks for the effort if you work on that. I tried to rebuild from scratch, but same result.

Do you have any update on this ? I could work from times to times to contribute to the project, but for now, I can't. I'd be glad to contribute.

Try these changes.
package.json
    "build": "rimraf dist && node tools/build.js",

tools\build.js
// Constants for running typescript commands
const TSC = 'node_modules\\.bin\\tsc.cmd';
const NGC = 'node_modules\\.bin\\ngc.cmd';

Hi @tja4472 , thanks. It did the trick!
I had no more errors for the npm run build. Then, I tried npm run test, and everything went well.
I think i'm now able to contribute, even on windows. Not sure how to propose PR because of the necessary changes to tools/build.js & package.json files. But anyway, i can try.


ddev launch: DDEV_PRIMARY_URL uses first hostname from alphabetically sorted list of hostnames

Describe the bug
The ddev launch command uses $DDEV_PRIMARY_URL to set the hostname to open in a browser. It seems that the list of hostnames is sorted alphabetically.
Expected behavior
I'd expect ddev to use the project name as DDEV_PRIMARY_URL.
To Reproduce
Create a project with this configuration:
APIVersion: v1.12.0
name: ekw
type: typo3
docroot: public
php_version: "7.3"
webserver_type: apache-fpm
router_http_port: "80"
router_https_port: "443"
xdebug_enabled: false
additional_hostnames:
- ckm
additional_fqdns: []
mariadb_version: "10.2"
nfs_mount_enabled: false
provider: default
use_dns_when_possible: true
timezone: Europe/Berlin

When you start ddev, you can already see the alphabetically sorted list of hostnames:
Successfully started ekw
Project can be reached at https://ckm.ddev.site https://ekw.ddev.site https://<IP_ADDRESS>:32772

If you hit ddev launch, https://ckm.ddev is opened instead of the expected https://ekw.ddev.site
Additional context
I've already discussed this issue with Randy on Slack:
https://typo3.slack.com/archives/C8TRNQ601/p1577731610020600
Seems that the hostnames are sorted here:
https://github.com/drud/ddev/blob/4d7cf50357e76654e37b6bd01549480047cd84e0/pkg/ddevapp/ddevapp.go#L1511
https://github.com/drud/ddev/blob/06754053ebecabe991fb25019eeb1a6fd8d85b81/pkg/ddevapp/config.go#L495

We should sort only the ones that come after the basic URL, not sort all together as equals.


SPAM

SPAM

I was referring to [Secure Multi-Party Computation (https://en.wikipedia.org/wiki/Secure_multi-party_computation) which in general is interested in methods/protocols that allow

parties to jointly compute a function over their inputs while keeping those inputs private.

Whether this feasible or desirable is not clear but can agree that it would be a stronger alternative to a privacy policy that would increase Kite's appeal to users, especially given the type of user it is meant to benefit.

All apologies, I though that this repo was for https://kite.com/
What was I thinking? Sorry, about all that.


TypeError: Cannot read property 'presets' of undefined

Hello, I receive this error message:
TypeError: Cannot read property 'presets' of undefined
With this challenge:
https://github.com/freeCodeCamp/freeCodeCamp/blob/master/curriculum/challenges/english/02-javascript-algorithms-and-data-structures/basic-data-structures/add-items-to-an-array-with-push-and-unshift.english.md
Solution:
function mixedNumbers(arr) {
// change code below this line
arr.push(7,'VIII',9);
arr.unshift('I',2,'three');
// change code above this line
return arr;
}
Link to page with problem:
https://www.freecodecamp.org/learn/javascript-algorithms-and-data-structures/basic-data-structures/remove-items-from-an-array-with-pop-and-shift
(Message: TypeError: Cannot read property 'presets' of undefined)

Browser Name: Chrome
Browser Version: Version 79.0.3945.130 (Official Build) (64-bit)
Operating System: macOS High Sierra 10.13.6

Please use our forum for debugging the code of your projects.  If the forum members determine there is nothing wrong with your code, you can request this issue to be reopened.  GitHub issues are meant for reporting bugs and not troubleshooting the code in your projects.
Thank you for understanding.

This isn't a problem with the user's code, when I use their code the tests pass.  It's probably related to #38077 since I changed how the presets were loaded.
@mikecullen76 are you still seeing this problem?

@ojeytonwilliams Thank you, the problem is now solved.


One or more points were written more frequently than the maximum sampling ...

I just set up stackdriver with prometheus in my kubernetes cluster. I have debug level logging turned on for the sidecar (it was very useful for debugging all the permissions issues I ended up running into). Now that the flow is working and sending data to stackdriver I'm noticing a lot of debug level messages in the logs that look like:
level=debug ts=2020-03-09T22:32:56.082Z caller=client.go:202 component=storage msg="Partial failure calling CreateTimeSeries" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: external.googleapis.com/prometheus/scrape_samples_post_metric_relabeling, Timestamps: {Youngest Existing: '2020/03/09-15:32:38.215', New: '2020/03/09-15:32:42.308'}}: timeSeries[0-31]"

Not just that specific metric, but many, many others. Any guidance on why this might be happening and how I can configure things to fix the problem (assuming this is in fact a problem).

I am getting the same error but seems that it happens sending some series to an existing metric:
level=warn ts=2020-05-14T09:01:32.691Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[0-73]"

I do not know how to debug/know for which metrics is throwing those errors.
Seems like the scrape_interval plays a major component for the issue, so for scrape_interval lower that 60s the error appears very frequently; for 60s the errors appears sometimes and for scrape_interval of 120s the error does not appear at all. Current quota limit for TimeSeries is 1metric/10 secods, but it seems currently in my project the limit is the old quota limit of 1metric/60 secods.
I would appreciate any help/progress on this issue.

@jcortejoso - I think the metric descriptor may still show up in other lines of the debug logs - in the log you provided, the message may be truncated. I have noticed before that some lines have the metrics name in the debug logs, sometimes they didn't show up.
It may be the logs are truncated, but it may also be bugs causing the metric name not logged.
See the debug log from the dopey - which has the actual metric name external.googleapis.com/prometheus/scrape_samples_post_metric_relabeling, and other debug information are similar to your logs.


Questions about evaluation.

Hi! I have 2 questions about evaluation. I'd be grateful for you help.

Where to set directory of dataset before evaluation?
When running to the following code, the pygame window crash and the program is stuck.

# leaderboard/team_code/lmdriver_agent.py: DisplayInterface.__init__
        self._display = pygame.display.set_mode(
            (self._width, self._height), pygame.HWSURFACE | pygame.DOUBLEBUF
        )

I tried to execute the following code separately, the pygame window continues to open.
import pygame
pygame.init()
pygame.font.init()
pygame.display.set_mode((1200, 900), pygame.HWSURFACE | pygame.DOUBLEBUF)

Thanks in advance!

Hi!

You can put the dataset anywhere you can access it. When you use the dataset to pretrain or finetune the model, the dataset path should be set to your dataset path.
Could you provide more details or the log? BTW, the two code blocks look similar? I can't understand the following context:

# leaderboard/team_code/lmdriver_agent.py: DisplayInterface.__init__

Hi!

The input to the model during evaluation does not come from the dataset you provided, but from the simulator. Is that right?
Sorry for not describing it clearly. Here is the detailed information about the problem I encountered when evaluation. Are there anything wrong with my operating procedures?

My Command:
First, run calar in one console.

bash carla/CarlaUE4.sh --world-port=2000

Second, run evaluation in another console.
export PT=2000

export CARLA_ROOT=carla
export CARLA_SERVER=${CARLA_ROOT}/CarlaUE4.sh
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla
export PYTHONPATH=$PYTHONPATH:$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg
export PYTHONPATH=$PYTHONPATH:leaderboard
export PYTHONPATH=$PYTHONPATH:leaderboard/team_code
export PYTHONPATH=$PYTHONPATH:scenario_runner

export LEADERBOARD_ROOT=leaderboard
export CHALLENGE_TRACK_CODENAME=SENSORS
export PORT=$PT # same as the carla server port
export TM_PORT=$(($PT+500)) # port for traffic manager, required when spawning multiple servers/clients
export DEBUG_CHALLENGE=0
export REPETITIONS=1 # multiple evaluation runs
export ROUTES=langauto/benchmark_long.xml
export TEAM_AGENT=leaderboard/team_code/lmdriver_agent.py # agent
export TEAM_CONFIG=leaderboard/team_code/lmdriver_config.py # model checkpoint, not required for expert
export CHECKPOINT_ENDPOINT=results/lmdrive_result.json # results file
#export SCENARIOS=leaderboard/data/scenarios/no_scenarios.json #town05_all_scenarios.json
export SCENARIOS=leaderboard/data/official/all_towns_traffic_scenarios_public.json
export SAVE_PATH=data/eval # path for saving episodes while evaluating
export RESUME=False

echo ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator.py
python3 -u  ${LEADERBOARD_ROOT}/leaderboard/leaderboard_evaluator.py \
--scenarios=${SCENARIOS}  \
--routes=${ROUTES} \
--repetitions=${REPETITIONS} \
--track=${CHALLENGE_TRACK_CODENAME} \
--checkpoint=${CHECKPOINT_ENDPOINT} \
--agent=${TEAM_AGENT} \
--agent-config=${TEAM_CONFIG} \
--debug=${DEBUG_CHALLENGE} \
--record=${RECORD_PATH} \
--resume=${RESUME} \
--port=${PORT} \
--trafficManagerPort=${TM_PORT}

Program is stuck, and the information is shown below.

leaderboard/leaderboard/leaderboard_evaluator.py:24: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
bc1
localhost 2000
bc2
bc3
leaderboard/leaderboard/leaderboard_evaluator.py:95: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(dist.version) < LooseVersion('0.9.10'):
bc4
/root/anaconda3/envs/lmdrive/lib/python3.8/site-packages/diffusers/models/cross_attention.py:30: FutureWarning: Importing from cross_attention is deprecated. Please import from diffusers.models.attention_processor instead.
  deprecate(
pygame 2.5.2 (SDL 2.28.2, Python 3.8.13)
Hello from the pygame community. https://www.pygame.org/contribute.html
bc5
bc

========= Preparing RouteScenario_3 (repetition 0) =========
> Setting up the agent

After checking, I find the program is stuck in the following position:

  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/leaderboard_evaluator.py(497)<module>()
-> main()
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/leaderboard_evaluator.py(488)main()
-> leaderboard_evaluator.run(arguments)
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/leaderboard_evaluator.py(423)run()
-> self._load_and_run_scenario(args, config)
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/leaderboard_evaluator.py(277)_load_and_run_scenario()
-> self.agent_instance = getattr(self.module_agent, agent_class_name)(args.agent_config)
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/leaderboard/autoagents/autonomous_agent.py(45)__init__()
-> self.setup(path_to_conf_file)
  /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/team_code/lmdriver_agent.py(176)setup()
-> self._hic = DisplayInterface()
> /media/samsung/samsung/mc.wei/code/LMDrive-main/leaderboard/team_code/lmdriver_agent.py(81)__init__()
-> self._display = pygame.display.set_mode(

I find that when executing this code  self._display = pygame.display.set_mode, a pygame window open but immediately collapsed. Afterwards, the program will remain stuck here.

Hi!

Yes. Our pipeline is based on the closed-loop and end-to-end setting. The evaluation needs to interact with the simulator in a real-time manner.
Sorry, I can't reproduce the error and I haven't encountered this problem. Did you run it on a docker/server environment? Maybe you can try a newer version of pygame. Or remove pygame.HWSURFACE | pygame.DOUBLEBUF when you initialize the display. Or comment on corresponding pygame codes to run the evaluation without the GUI.

Thanks for your fast response. I will try according to your recommend.

Thanks for your fast response. I will try according to your recommend.

Hi, I am meeting the same issue, were you able to solve it?


Allow auto vivification for nested structured types

The idea is that we can use Crystal type system to support auto vivification of Hashes and Arrays like we have in perl.
Consider this:
@update_tm_to_product_property = Hash(Time,Hash(Int32,Hash(Int32,Set(Int32)))).new { |h,k|
   h[k] = Hash(Int32,Hash(Int32,Set(Int32))).new { |h,k|
      h[k] = Hash(Int32,Set(Int32)).new { |h,k|
         h[k] = Set(Int32).new
      }
   }
}

Now this clumsy way is the only to allow auto create of nested structures to be able write something like @update_tm_to_product_property[updated_at][group_id][product_id].add(property_id)
But look at type definition: Hash(Time,Hash(Int32,Hash(Int32,Set(Int32))))
We already have all exact information about the whole structure. Why not to allow auto vivification of nested structures is default behaviour for such types? Just to be able to write
@update_tm_to_product_property = Hash(Time,Hash(Int32,Hash(Int32,Set(Int32)))).new
and use it as I gave an example.
Why not ?

The defaults values for keys depends too much on the operations to be done afterwards: either T.new, T.zero, or 1 (if applied) might be useful default values.
I don't think it's an idiom so common to be in the std. Do you know any language that has that deep vivification?
Either way, you can refactor the code to reduce the type annotations:
h0 = Hash(Int32, Set(Int32)).new { |h, k| h[k] = Set(Int32).new }
h1 = Hash(Int32, typeof(h0)).new { |h, k| h[k] = h0 }
hash = Hash(Time, typeof(h1)).new { |h, k| h[k] = h1 }

t0 = Time.now
hash[t0][1][2] << 3
pp hash[t0][1][2]

Or even create a macro to do it for you
macro auto_vivify_hash(*types)
  %v{2} = Hash({{types[-2]}}, {{types[-1]}}).new { |h, k| h[k] = {{types[-1]}}.new }
  {% for i in 3..types.size %}
    %v{i} = Hash({{types[-i]}}, typeof(%v{i - 1})).new { |h, k| h[k] = %v{i - 1} }
  {% end %}
  %v{types.size}
end

hash = auto_vivify_hash(Time, Int32, Int32, Set(Int32))

t0 = Time.now
hash[t0][1][2] << 3
pp hash[t0][1][2]

@bcardiff wow, that's the first time I'm seeing %v{2} notation. Does {} denote anything else than part of the variable name?

%var ensure a fresh variable is used when expanding them. and the {index} sufix is a handy way to create temp variables across loop iterations for example and do trickery things 🎩  🐇  .
As usual, the first step is to code things in a plain way and get a pattern.

Auto vivification should work only for Structures, like Hashes and Arrays inside Hashes and Arrays when you use another level of subscription [] of [].
my_hash[k1][k2] - this should create some structure at my_hash[k1]
In perl5 and perl6 we definitely have autofification for Hashes and Arrays. In Perl there is 2 different ways of subscriptions: [] - for arrays, {} - for hashes. So it is always distinguishable what neccessary to vivify - array o hash:
my %h;
%h{"k1"}[5] = 6

will create array at %h{"k1"}
my %h;
%h{"k1"}{5} = 6

will create another hash at %h{"k1"}
It also works for deep subscriptions %h{"k1"}[5]{"k2"}[6]++
So in Crystal we also have enough information about what nested structure should be auto vivified because we have full type definition.
So this is special, not universal case - Not T.zero, 1 or something else possible when we want to have [] of it, and type definition point us that there is should be Array or Hash.
Of course it is possible to solve using macros. But the macro example you gave is quite difficult to write for average Crystal developer and, I think, should be part of standard lib.

Sometimes you have to write a bit more in Crystal, and that's fine.


Create internal update system

We need an internal update system to keep everyone up-to-date with the latest version of Spotio.
I have absolutely zero experience with anything like that so I'd appreciate all the help I can get!!
The ideal functionality would be something like this:

When Spotio is launched, check to see if an update has been posted on Spotify's server.
If it has, prevent the default app's update and alert me that there's an update
I'll update the Spotio code and push a new release
Then the Spotio app will download the latest release

Again, I have NO idea how this kind of thing works so I'm just spitballing, but this feature is a MUST HAVE. Anyone up for the challenge??

Not an expert in this either but have seen some suggestions to a feature for this on wmail
Some comments made suggestions to:

sparkle-project
electron-updater

The overall conclusion was cost of implementing a update/release system & signing the application for any given OS implementation.
Hope this helps.

Last night, I had a quick look for the usual suspects in the Javascript files inside the OS X app. But I was unable to find any references to an auto-updater in the Javascript. Ideally that would be the best place to hook in a Spotio updater.
So my current guess is, that the update functionality is baked inside of the binary. :(
I hope there are some Javascript API calls available in CEF (Chromium Embedded Framework), that trigger an event (or a Javascript callback) when an update is available for downloading. I will look into this and report back.

Thanks for the help @avelis and @sandervankasteel!! Yeah I'm assuming they're doing updates via the binary, so it might not be possible to stop them entirely from within the app.
I think a good approach might be to create an external update helper (maybe with Sparkle or electron? I don't have any experience with either but I can look into them) that blocks outgoing connections to spotify's update server and handles updates on its own whenever it sees an update available on the server

I'm not sure how that'd work logistically yet but in theory it'd be possible. I'm also very worried about how to diff Spotio's assets with Spotify's assets when an update occurs since Spotio's asset codebase is beginning to diverge from Spotify's default assets. Anyone have thoughts on this?

Bit of a shot in the dark here.
Would it not be possible to wrap the Spotify binary in an electron app that would check for new versions with a checksum and apply the needed skinning?
Then you could auto update changes to the skin and make sure that Spotify is always running on the latest version.

My "attack" plan on this is based around a theory.
The rough theory : I doubt that Spotify will send a complete new binary DMG file to the client when an update is available. My guess is that they let the client fetch a zip / tar.gz / gzip / bz2 file with only the changes based on their current version. If that the case, they will probably have a patch / replace function in their big binary blob that handles dealing with that zip / tar.gz / gzip / bz2 file and checksumming any files.
If my theory is right; that means the Spotify.app directory (atleast on OS X) will never be completely replaced and that means we could theoretically(!) inject / load our own Javascript, which could handle our own internal updating.
My tactic on the logistics of this, use GitHub releases and let our 'versions' match the versions of Spotify. So the client does a call to Github asking for the last release, checks if the latest version on Github matches the client's local version and applies patches locally if necessary.
Github API call for getting releases : https://developer.github.com/v3/repos/releases/
But if I am wrong .....  I need some more time to think about a solution.
@pthm regarding your idea.. I like your idea, but I'm hesitant in using a solution like that because it adds an extra wrapper around an already kind of bloated application.

Addition to my previous theory: this doesn't apply to to Linux, since it relies on the package manager of the local machine. The situation is unknown to me on Windows.
What is also an option is that we turn Spotio into a standalone patcher application using Electron or CEF and logistically we could still use the same method I described earlier with matching our versions with the Spotify versions. And figure out that way if we need to patch the local Spotify installation.

I just got home started Spotify and was able to see this screen;

From that I can only conclude that Spotify does some internal checksumming to check of the files have been tampered with. This is something we should take into account.

I'm thinking that that error is caused by having multiple versions of the Spotify app signed in on your computer. Not positive. I'm going to try to recreate their .spa file format (I believe it's just a .zip renamed to .spa, but when I tried doing that it didn't work correctly)

@sandervankasteel I think your attack plan sounds good and logistically possible, we'll have to play around with some code and see if we can hack this together as an MVP before we can move forward with a release version though.
Another feasible (?) option would be to inject a helper that replaces all text matching a string in the updated assets with Spotio's own local strings (i.e. #333333 replaced with #ffffff, etc.). This would pose the issues that: there will be no fine controls on our end to control the visual updates; if we change one instance of the string, we will likely be changing them all; it may be very unpredictable as a process since spotify could add rules to the css that we did not foresee while developing the update system and this will break the app's design.
I'd ultimately like to pick the version of the update system that is most usable by everyday Spotify listeners (i.e. non-designer/developers), but if we must sacrifice a subset of users in order to satisfy a larger portion, I'm okay with that.

Sorry if my thought process wasn't clear in that last comment; it's late and I'm exhausted. I'll clarify if anyone needs it.

I wanna see if I can build a MVP of an Electron / CEF patcher application, this weekend. It's definitely gonna be "quick 'n dirty", barely functioning and will most likely only support Linux and Mac OS for the moment. But should give a clear direction if it feasible to go into this direction.

Thanks, go for it! Just a basic proof of concept would be cool and if it's possible we can run with it and I can work on the Windows support and clean it all up based on your MVP. I wish I could put more time into the project right now but I'm super busy with school

@devinhalladay Right now my school hasn't started yet, so I have some spare time left! Once my school starts I will have the same problem ;)

So, I think scripting it would be the best option (springing off of the conversation in #23)—my thought currently is that this is what we need:

helper app (cross-platform, obviously), to block connections to Spotify's update servers
an option (developer mode, not sure how this would work yet) so that I can bypass the connection block and test new updates on my machine
this helper app will also act as an updater
so, when I push a new release, the helper app will work relatively the same as any other update system: it will pull down the zip, replace the files in your local spotify app with the updated files
relaunch the spotify app, and then keep listening for new updates

Since this needs to be cross-platform, I'm thinking we need to find a way to work with Electron, unless we can find developers who are willing to contribute each system's update app individually according to some code standards to keep them unified.

Any thoughts on that? It's the only way I can think of doing this pragmatically—we need to block Spotify's update servers, one way or another, in order to keep our updates flowing smoothly, and we also need to have our own streaming update system in place. This will likely mean that I need to restructure the build system in order to more readily make updated assets (.spa's) available to the helper app so that it can simply unzip the release to a temporary directory, copy over the files to the appropriate Apps directory, then delete the temp dir. That's the easy part, though.

Looks like we can probably just use this, but Linux will have to fall by the wayside<EMAIL_ADDRESS>

Starting work on this now, using Electron.


call reencrypt in batch to reveal all players' results

The first game had 22 players, the winning answer was 16, and there was one winner.
We have a reencrypt function to reveal each player's vote, ideally we can do it in batch so a bar chart can be generated.

once batch function is done, it's better to have a front-end button to let user reveal their own results. This functionality can also apply to other games.

TBH, I think the following
function reencryptAny(address player, bytes32 publicKey, bytes calldata signature)
        public
        view
        onlySignedPublicKey(publicKey, signature)
        gameEnded
        returns (bytes memory)
    {
        return TFHE.reencrypt(entries[player], publicKey, 0);
    }

is uneccessary, because at the end of the game, simply a decrypt is enough

We could add a decrypt function at the end for trusted public reveals but I would like to keep the reencyrpt because it is gasless. So if we call this on the front end, we can create a smoother UX without forcing the user to make an extra transaction.

Pushed PR, PTAL @0xAkrasia

PR is merged