Fix unpadding input with padding mask compute error

Fix unpadding input with padding mask compute error

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.wangboyun seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Fix custom hit testing

What does the pull request do?

Fixes custom hit-testing broken in https://github.com/AvaloniaUI/Avalonia/pull/5923
What is the current behavior?

Custom hit testing when custom title bar is used not working.
What is the updated/expected behavior with this PR?

Fixes custom hit-testing.
How was the solution implemented (if it's not obvious)?

Check for correct interface.
Checklist

[ ] Added unit tests (if possible)?
[ ] Added XML documentation to any related classes?
[ ] Consider submitting a PR to https://github.com/AvaloniaUI/Documentation with user documentation

Breaking changes

Obsoletions / Deprecations

Fixed issues

Still not sure this is right, and I've been unable to test it: the description says "Custom hit testing when custom title bar is used not working." but when I run ControlCatalog and go to "Dialogs -> Decorated Window" the custom title bar seems to work fine?
First of all how do I repro the problem in order to verify the fix?

@grokys repro:

Run ControlCatalog.NetCore
Got to Window Customizations page
Check Extend Client Area to Decorations and Prefer System Chrome
Try dragging window by titlebar


Fix API post/boost IDs

Mastodon might return IDs in the API as strings but it intends for them to be sortable by ID (https://docs.joinmastodon.org/api/guidelines/#id) and some clients rely on this particular attribute of them (for example, Tusky), so we need for IDs in the API to behave appropriately.
Currently, we use 1234 for posts and interaction-1234 for boosts which breaks this.
We have two options:

Dynamically translate to/from an extended ID format in the API layer
Move Post and PostInteraction to use a snowflake-style ID that are distinct from each other

Snowflake Ids were implemented with https://github.com/jointakahe/takahe/commit/ecec5d6c0a4df7b43c67e1fe4b7503d5962e4426


fix: NO-JIRA library not working

Fix dialtone mono-package not working
Obligatory GIF (super important!)

:hammer_and_wrench: Type Of Change
These types will increment the version number on release:

[x] Fix

:book: Description

Removed icons lazy-loading for now, we're having a lot of issues related to it.
Tree-shaking and size improvements have been maintained
Tested on ubervoice via pnpm pack and worked perfectly.

:bulb: Context
A lot of issues updating to latest dialtone on dp, dpm and dp 2.0
:pencil: Checklist
For all PRs:

[x] I have ensured no private Dialpad links or info are in the code or pull request description (Dialtone is a public repo!).
[x] I have reviewed my changes.
[x] I have considered the performance impact of my change.

:crystal_ball: Next Steps
Release and update on products

Merging this to make a release so people don't install the broken version


Peter_frank_wayne_hello_world_graph

you need to .gitignore the build directory...
do that because it is too messy right now and i will look again.

also it seems like there are too many commits.
it does not have to be 1 -- you can organize it in a few, but make it well organized.
you can squash some as needed


Pinned ivadomed dependencies causing downstream dependency conflicts for SCT

In https://github.com/ivadomed/ivadomed/pull/759, ivadomed began pinning dependencies to specific versions.
Unfortunately, as a result, these newer, stricter requirements have made ivadomed incompatible with SCT's requirements. And, since then, SCT's installer has been quietly installing ivadomed 2.7.4 (the last version before #759) and ignoring future releases.
Specifically, the conflict occurs with this pinned dependency:
https://github.com/ivadomed/ivadomed/blob/d6fe24db4d6a4319c57802b7a6d68e17f6d8b9e0/requirements_common.txt#L20
Since SCT relies on an older version of Tensorflow (and thus an older version of Tensorboard), the following conflict occurs when I try and force the latest version of ivadomed:
 The conflict is caused by:
    ivadomed 2.9.1 depends on tensorboard~=2.4
    tensorflow 1.15.5 depends on tensorboard<1.16.0 and >=1.15.0
    ivadomed 2.9.1 depends on tensorboard~=2.4
    tensorflow 1.15.4 depends on tensorboard<1.16.0 and >=1.15.0
    ivadomed 2.9.1 depends on tensorboard~=2.4
    tensorflow 1.15.3 depends on tensorboard<1.16.0 and >=1.15.0
    ivadomed 2.9.1 depends on tensorboard~=2.4
    tensorflow 1.15.2 depends on tensorboard<1.16.0 and >=1.15.0
    ivadomed 2.9.1 depends on tensorboard~=2.4
    tensorflow 1.15.0 depends on tensorboard<1.16.0 and >=1.15.0

Since SCT is somewhat stuck with its dependency on Tensorflow 1.15 for the time being, would it be possible for ivadomed to try loosening up its dependencies? (See also, this comment I made in the original issue: https://github.com/ivadomed/ivadomed/issues/501#issuecomment-719990834)

This is becoming quite urgent because it blocks us on a project we need to deliver (this one)


Changed the keys for association errors to be symbol, rather than string

Hi. Thanks for the PR.
This is a 1:1 code with Rails codebase, so I cannot merge it until it's changed there as well.
Can you describe what problem it solves?

@morgoth
ohhh okay didn't know that.
I was adding a method on ActiveModel::Error, and just noticed that i couldn't access the error detail when it's the one for association.
I thought it was just a mistake. I'll close this PR.
thanks.


Desktop icons hidden AFTER AND BEFORE UNINSTALL

System information:

Windows version: Windows 11 Home
Cairo version: idk
32-bit or 64-bit: 64 bit
Removed my desktop icons. The files are still there, but the desktop won't show them. And it is uninstalled, please help. Screenshot:
 desktop is above, if you were wondering.
Oh. and wallpapers are here. https://www.minecraft.net/en-us/collectibles/minecraft-collection Note: its the beach cabin wallpaper.

Right-click your desktop > View > select "Show desktop icons" and they should appear!


Detach is not working

Hi, I was trying to do very simple, detatch an element, but after do it, and check if was removed maybe seems not to be the case? this is because I'm also using eq to compare the result, just here is something wrong.
use kuchiki::traits::*;

fn main() {
    let html = kuchiki::parse_html().one(r#"

"#);
    let html2: kuchiki::NodeRef = html.clone();

    if let Some(element) = html.select_first("table").ok() {
        let as_node = element.as_node();
        as_node.detach();
    }

    let e = html.eq(&html2);
    println!("{e}");
}

It returns true... when they are not equals, or maybe I did something bad?
Thx!

Oks, there is other issues, but this is not the right one.


Click Sound when Resetting

I found that if I don't do anything for a bit and then reset I can hear a click sound, very minor but just something to note on

I think this is just a thing you only notice because no reset sounds tbh

yeah

Not really fixable


[Bug]: Importing a Keystore Documentation is Outdated (wrong)

Describe the bug
https://docs.chain.link/docs/miscellaneous/#importing-a-keystore shows that the command to import a keystore is "chainlink local import ./keystore.json"
However, the command is now:
chainlink keys eth import --oldpassword oldpasswordcredsfile keystore.json
To Reproduce

Connect to CL node docker container and run command listed in docs
Get back (my keystore file is call privatekey.json):
root@ce929a0bff50:/chainlink# chainlink local import privatekey.json
2022-06-23T17:57:51.000Z [DEBUG] Initializing API credentials                       cmd/client.go:636       file=/chainlink/apicredentials
No help topic for 'import'
root@ce929a0bff50:/chainlink#

URLs
https://docs.chain.link/docs/miscellaneous/#importing-a-keystore
Expected behavior
root@ce929a0bff50:/chainlink# chainlink keys eth import --oldpassword apicredentials privatekey.json
2022-06-23T18:00:13.563Z [DEBUG] Initializing API credentials                       cmd/client.go:636       file=/chainlink/apicredentials
parseResponse error: Error; key with ID 0x345aF178A3D894815EEC819A4E0b7C68E2584468 already exists
except instead of the key already exsting, it would import 🤷 (this has been tested by a customer)
Additional context
No response

Hi @SeaAnenome
sorry for the delay and many thanks for reporting the issue.


How to use XGBoost PySpark API with MLeap?

Problem description:
We were able to serialize the XGBoost model with MLeap using the older PySpark API (https://github.com/dmlc/xgboost/pull/4656) as shown below:
import mleap.pyspark
from mleap.pyspark.spark_support import SimpleSparkSerializer

trans_model = model.transform(df)
local_path = "jar:file:/tmp/pyspark.model.zip"
model.serializeToBundle(local_path, trans_model)

But We are not able to do with official PySpark API (https://xgboost.readthedocs.io/en/stable/tutorials/spark_estimator.html) support. Is there anything flags that I need to use to make this work?
Steps Followed:
Download spark
wget https://dlcdn.apache.org/spark/spark-3.3.3/spark-3.3.3-bin-hadoop3.tgz
tar -xvf spark-3.3.3-bin-hadoop3.tgz
Download xgboost jars
wget https://repo1.maven.org/maven2/ml/dmlc/xgboost4j_2.12/1.7.3/xgboost4j_2.12-1.7.3.jar
wget https://repo1.maven.org/maven2/ml/dmlc/xgboost4j-spark_2.12/1.7.3/xgboost4j-spark_2.12-1.7.3.jar
Build MLeap fat jar :
https://github.com/combust/mleap/blob/master/mleap-databricks-runtime-fat/README.md
git clone --recursive https://github.com/combust/mleap.git
cd mleap
git checkout tags/v0.22.0
sbt mleap-databricks-runtime-fat/assembly
cp mleap-databricks-runtime-fat/target/scala-2.12/mleap-databricks-runtime-fat-assembly-0.22.0.jar ../spark-3.3.3-bin-hadoop3/jars
Install python requirements
pip install mleap==0.22.0
pip install xgboost==1.7.3
pip install pyarrow
Running the example
cd ../spark-3.3.3-bin-hadoop3
./bin/spark-submit example.py
error log
23/09/13 22:06:29 INFO CodeGenerator: Code generated in 25.565617 ms
+---------+----+-------+--------+----------+------+--------+-----+------------+----+----+-----+--------------------+-------------+----------+-----------+
|dayOfWeek|hour|channel|platform|deviceType|adUnit|pageType|  zip|advertiserId|taxA|taxB|label|            features|rawPrediction|prediction|probability|
+---------+----+-------+--------+----------+------+--------+-----+------------+----+----+-----+--------------------+-------------+----------+-----------+
|        7|  20|      3|       6|         1|    10|       3|53948|      245351|   1|   2|    1|[7.0,20.0,3.0,6.0...|   [-0.0,0.0]|       0.0|  [0.5,0.5]|
|        7|  20|      3|       6|         1|    10|       3|53948|      245351|   1|   2|    1|[7.0,20.0,3.0,6.0...|   [-0.0,0.0]|       0.0|  [0.5,0.5]|
|        7|  20|      1|       6|         1|    10|       3|53948|      245351|   1|   2|    0|[7.0,20.0,1.0,6.0...|   [-0.0,0.0]|       0.0|  [0.5,0.5]|
|        7|  20|      1|       6|         1|    10|       3|53948|      245351|   1|   2|    0|[7.0,20.0,1.0,6.0...|   [-0.0,0.0]|       0.0|  [0.5,0.5]|
|        5|  20|      1|       6|         1|    10|       3|53948|      245351|   1|   2|    0|[5.0,20.0,1.0,6.0...|   [-0.0,0.0]|       0.0|  [0.5,0.5]|
|        5|  20|      3|       6|         1|    10|       3|53948|      245351|   1|   2|    1|[5.0,20.0,3.0,6.0...|   [-0.0,0.0]|       0.0|  [0.5,0.5]|
+---------+----+-------+--------+----------+------+--------+-----+------------+----+----+-----+--------------------+-------------+----------+-----------+
Traceback (most recent call last):
File "/Users/s0a018g/local_setup/spark-3.3.3-bin-hadoop3/bin/code1.py", line 47, in
model.serializeToBundle(local_path, predictions)
File "/Users/s0a018g/opt/anaconda3/lib/python3.8/site-packages/mleap/pyspark/spark_support.py", line 25, in serializeToBundle
serializer.serializeToBundle(self, path, dataset=dataset)
File "/Users/s0a018g/opt/anaconda3/lib/python3.8/site-packages/mleap/pyspark/spark_support.py", line 42, in serializeToBundle
self._java_obj.serializeToBundle(transformer._to_java(), path, dataset._jdf)
File "/Users/s0a018g/local_setup/spark-3.3.3-bin-hadoop3/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 363, in _to_java
AttributeError: 'SparkXGBClassifierModel' object has no attribute '_to_java'
23/09/13 22:06:30 INFO SparkContext: Invoking stop() from shutdown hook
example.py
Thanks @agsachin for creating the instructions that can be easily reproduced on Mac.

@WeichenXu123 Any insights on this issue? Please help us.

@venkatacrc MLeap serializeToBundle work only with java objects.
The latest XGBoost Estimator is a pure python implementation instead a Java Estimator
OLD API
class XGboostEstimator(JavaEstimator, XGBoostReadable, JavaMLWritable, ParamGettersSetters):

vs NEW API
class _SparkXGBEstimator(Estimator, _SparkXGBParams, MLReadable, MLWritable):

@austinzh Will there be any support for the Java like the old API in the future?

If we are going to support pure python implementation of _SparkXGBEstimator that will be a lot of groundwork.

We need to re-implement SimpleSparkSerializer to work in Python instead of JVM, for that we can use MLeapSerializer
We need to implement an Serializer that work with Python XGboost Model like sklearn


Move to a Mithril org

Im just making this issue to see what people, and of course Leo, think. It has come up a couple of times from people in issues and on the gitter channel and it is something I have thought for a while.
I think that the increased professional perception created by moving this repo into an organisation is, alone, enough of a reason to do so but obviously this is Leo's project.
I believe there is a recurring theme at the moment around 'how mithril can be a success' or other things of a similar sentiment and just wanted to gauge what people thought of this simple change which I think could be a move in the right direction

One does exist, but there would definitely need to be some work done.
https://github.com/mithriljs
I'm happy to help however I can with this project.

@lhorie I think it might be a good idea to move this repo to the MithrilJS org. Looks oddly barren ATM, so I feel it could use something.

@orbitbot To cover some of those points, in no particular order:

website could live under a meaningful repository name that doesn't cause confusion

Or maybe, just relocate it to the gh-pages branch of the core repo. If necessary, here's the script I use to deploy my personal site to gh-pages, deploying from dist. The catch with my approach is that I have to keep the generated files checked into source control. The docs can be easily kept in sync like the bundles already are, so that's not a problem. Alternatively, you could use a fixed fork (security hole) of this utility.

breaks gitter chat, at least temporarily

In my personal experience Gitter does redirect, too. (It redirected after me renaming a repo, so I expect it would also redirect migrating to an org.)

doesn't really accomplish anything other than a veneer unless some operational changes

We've already got some areas where we could use some of those changes. In particular, we could totally use a separate discussion repo, where we can have centralized discussion about tooling. That would be much easier to do via a dedicated org. Oh, and it'd make it much easier to address the more opinion-driven issues and ideas from the more opinionated people.

Leo might not want to?

Not sure if he's actually weighed in much, and from the little he's shared about his opinions on the matter, he seems almost as neutral as you could get. Oh, and he's totally a BDFL.

also regarding previous, maybe there's not a defined ambition with mithril [...]

Mithril has never really had much of a roadmap. It's mainly just been a small, mildly opinionated UI framework that's just grown organically. If anything, that's been the primary ambition, a small framework that's highly pragmatic and just gets out of your way of making things.

org might potentially sprout a few more related repos

Or, even better, moving some of the most popular repos (e.g. mithril-node-render) under the umbrella, so it remains logically closer to Mithril itself, even if it's still community-maintained. To cite some precedent, Redux itself is under the reactjs org, despite being community-maintained. So I'm not as concerned. For what it's worth, it's not that hard to only subscribe to some repos within an org, the ones you actually care enough about to interact with.

but can also see that it really doesn't have any other impact than a cosmetic change

It will have a bigger effect than merely cosmetic changes - it's easier to manage things when they're better centralized, and it'll help for the SEO of all under the umbrella.

We need to have @lhorie a lot more involved with this discussion, though. So far, his answers have been really inconclusive, and he's the deciding factor on this one.

The org would be a plus for us, it's just one more thing that helps sell-through mithril as a viable framework to certain clients.

@orbitbot You pretty much hit the nail on the head. The main difference in moving to an organization is that the process would change. It has very little to do with where a single repository lives, so much as the process governing the community in general. Node's migration from a collection of various projects to a centralized organization (nodejs)/foundation (Node.js Foundation) is a very well documented example of how moving to a centralized organization with a dedicated process changes things dramatically, when you look beyond the veneer of it being in nodejs/node instead of joyent/node.
It brings a very big sense of maturity to the project, both to those working on the project itself and to those looking in from the outside. Someone unfamiliar would read creationix/nvm as some side project somebody wrote one day and facebook/react as some random project a company decided to open source. But it'd be way easier to see nodejs/node as a truly serious runtime, and mochajs/mocha as a serious test runner. Human psychology is at play here, because the former two are things that seem tied to an individual or company whose main product most definitely isn't that project, and the latter two are clearly branded as not merely a project, but an integral intersection of an entire community that have themselves formed around that project.
And I can tell you that when a project appears mature with a clearly vibrant community, that sells on its own, both to technical and non-technical people.

Since there's a few people who have :+1: the OP, maybe they would like to pitch in as well?

I'm actually one of them. :wink:

Closing, since this issue has not been updated in over a month, and typically, issues inactive for that long do not usually produce any action. If you feel something in Mithril needs added or changed, please file a new issue.


added mock

Codecov Report

Merging #49 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #49   +/-   ##
=======================================
  Coverage   93.42%   93.42%
=======================================
  Files           5        5
  Lines        1292     1292
=======================================
  Hits         1207     1207
  Misses         85       85

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a694341...9cc85b3. Read the comment docs.


Add implementation of extension creation/updating/adding/restoring

This change adds a NuGet backed way of creating/adding/restoring extensions.
NOTE: With this change, you can add/remove/list/use extensions. However, in order to restore on demand things that are currently synchronous need to be made async. That will be done in a subsequent change to minimize the amount of churn.

This change adds a NuGet backed way of creating/adding/restoring extensions.
With this change, you can add/remove/list/use extensions. However, in order to restore on demand things that are currently synchronous need to be made async. That will be done in a subsequent change to minimize the amount of churn.


Issue with running ChatUI (Space) with a fine-tuned model

I am trying to follow the example form here https://huggingface.co/blog/Llama2-for-non-engineers, I successfully manged to train llama2 7B model. but when I am trying to attache that model to a ChatUI. I get the following error:
===== Application Startup at 2023-11-24 00:27:54 =====
2023-11-24T00:40:22.465175Z  INFO text_generation_launcher: Args { model_id: "roozbeh397/ajpc-m6al-a51a-0", revision: None, validation_workers: 2, sharded: None, num_shard: Some(1), quantize: None, dtype: None, trust_remote_code: true, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 2048, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "r-roozbeh397-armhubbot-87ryh4wu-37f13-fbm2p", port: 8080, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }
2023-11-24T00:40:22.465224Z  WARN text_generation_launcher: trust_remote_code is set. Trusting that model roozbeh397/ajpc-m6al-a51a-0 do not contain malicious code.
2023-11-24T00:40:22.465299Z  INFO download: text_generation_launcher: Starting download process.
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to <IP_ADDRESS> port 8080: Connection refused
Warning: Transient problem: connection refused Will retry in 10 seconds. 60
Warning: retries left.
{"t":{"$date":"2023-11-24T01:40:22.480+01:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":17},"outgoing":{"minWireVersion":6,"maxWireVersion":17},"isInternalClient":true}}}
{"t":{"$date":"2023-11-24T01:40:22.482+01:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2023-11-24T01:40:22.482+01:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2023-11-24T01:40:22.484+01:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","namespace":"config.tenantMigrationDonors"}}
{"t":{"$date":"2023-11-24T01:40:22.484+01:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","namespace":"config.tenantMigrationRecipients"}}
{"t":{"$date":"2023-11-24T01:40:22.484+01:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"ShardSplitDonorService","namespace":"config.tenantSplitDonors"}}
{"t":{"$date":"2023-11-24T01:40:22.484+01:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
{"t":{"$date":"2023-11-24T01:40:22.484+01:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":6,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"r-roozbeh397-armhubbot-87ryh4wu-37f13-fbm2p"}}
{"t":{"$date":"2023-11-24T01:40:22.484+01:00"},"s":"W",  "c":"CONTROL",  "id":20720,   "ctx":"initandlisten","msg":"Memory available to mongo process is less than total system memory","attr":{"availableMemSizeMB":14336,"systemMemSizeMB":15802}}
{"t":{"$date":"2023-11-24T01:40:22.484+01:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"6.0.11","gitVersion":"f797f841eaf1759c770271ae00c88b92b2766eed","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2023-11-24T01:40:22.484+01:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
{"t":{"$date":"2023-11-24T01:40:22.484+01:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{}}}
{"t":{"$date":"2023-11-24T01:40:22.485+01:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=6656M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],"}}
{"t":{"$date":"2023-11-24T01:40:22.809+01:00"},"s":"I",  "c":"STORAGE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":324}}
{"t":{"$date":"2023-11-24T01:40:22.809+01:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2023-11-24T01:40:22.813+01:00"},"s":"W",  "c":"CONTROL",  "id":22120,   "ctx":"initandlisten","msg":"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted","tags":["startupWarnings"]}
{"t":{"$date":"2023-11-24T01:40:22.813+01:00"},"s":"W",  "c":"CONTROL",  "id":22140,   "ctx":"initandlisten","msg":"This server is bound to localhost. Remote systems will be unable to connect to this server. Start the server with --bind_ip  to specify which IP addresses it should serve responses from, or with --bind_ip_all to bind to all interfaces. If this behavior is desired, start the server with --bind_ip <IP_ADDRESS> to disable this warning","tags":["startupWarnings"]}
{"t":{"$date":"2023-11-24T01:40:22.814+01:00"},"s":"W",  "c":"CONTROL",  "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":524288,"recommendedMinimum":1677720,"maxConns":838860},"tags":["startupWarnings"]}
{"t":{"$date":"2023-11-24T01:40:22.814+01:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"admin.system.version","uuidDisposition":"provided","uuid":{"uuid":{"$uuid":"c166e3d5-1d75-4470-8810-82f79f81471f"}},"options":{"uuid":{"$uuid":"c166e3d5-1d75-4470-8810-82f79f81471f"}}}}
{"t":{"$date":"2023-11-24T01:40:22.818+01:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"c166e3d5-1d75-4470-8810-82f79f81471f"}},"namespace":"admin.system.version","index":"id","ident":"index-1-7274828463766787221","collectionIdent":"collection-0-7274828463766787221","commitTimestamp":null}}
{"t":{"$date":"2023-11-24T01:40:22.818+01:00"},"s":"I",  "c":"REPL",     "id":20459,   "ctx":"initandlisten","msg":"Setting featureCompatibilityVersion","attr":{"newVersion":"6.0"}}
{"t":{"$date":"2023-11-24T01:40:22.818+01:00"},"s":"I",  "c":"REPL",     "id":5853300, "ctx":"initandlisten","msg":"current featureCompatibilityVersion value","attr":{"featureCompatibilityVersion":"6.0","context":"setFCV"}}
{"t":{"$date":"2023-11-24T01:40:22.818+01:00"},"s":"I",  "c":"NETWORK",  "id":4915702, "ctx":"initandlisten","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":17},"outgoing":{"minWireVersion":6,"maxWireVersion":17},"isInternalClient":true},"newSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":17,"maxWireVersion":17},"outgoing":{"minWireVersion":17,"maxWireVersion":17},"isInternalClient":true}}}
{"t":{"$date":"2023-11-24T01:40:22.818+01:00"},"s":"I",  "c":"NETWORK",  "id":4915702, "ctx":"initandlisten","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":17,"maxWireVersion":17},"outgoing":{"minWireVersion":17,"maxWireVersion":17},"isInternalClient":true},"newSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":17,"maxWireVersion":17},"outgoing":{"minWireVersion":17,"maxWireVersion":17},"isInternalClient":true}}}
{"t":{"$date":"2023-11-24T01:40:22.818+01:00"},"s":"I",  "c":"REPL",     "id":5853300, "ctx":"initandlisten","msg":"current featureCompatibilityVersion value","attr":{"featureCompatibilityVersion":"6.0","context":"startup"}}
{"t":{"$date":"2023-11-24T01:40:22.818+01:00"},"s":"I",  "c":"STORAGE",  "id":5071100, "ctx":"initandlisten","msg":"Clearing temp directory"}
{"t":{"$date":"2023-11-24T01:40:22.818+01:00"},"s":"I",  "c":"CONTROL",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
{"t":{"$date":"2023-11-24T01:40:22.818+01:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
{"t":{"$date":"2023-11-24T01:40:22.819+01:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.startup_log","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"9855b4a2-03a0-429e-a1f1-fe9398cf50bd"}},"options":{"capped":true,"size":10485760}}}
{"t":{"$date":"2023-11-24T01:40:22.823+01:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"9855b4a2-03a0-429e-a1f1-fe9398cf50bd"}},"namespace":"local.startup_log","index":"id","ident":"index-3-7274828463766787221","collectionIdent":"collection-2-7274828463766787221","commitTimestamp":null}}
{"t":{"$date":"2023-11-24T01:40:22.823+01:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigReplicationDisabled","oldState":"ConfigPreStart"}}
{"t":{"$date":"2023-11-24T01:40:22.823+01:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
{"t":{"$date":"2023-11-24T01:40:22.824+01:00"},"s":"I",  "c":"CONTROL",  "id":20712,   "ctx":"LogicalSessionCacheReap","msg":"Sessions collection is not set up; waiting until next sessions reap interval","attr":{"error":"NamespaceNotFound: config.system.sessions does not exist"}}
{"t":{"$date":"2023-11-24T01:40:22.825+01:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"LogicalSessionCacheRefresh","msg":"createCollection","attr":{"namespace":"config.system.sessions","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"2ee0d9ec-bc3f-44b3-ac7e-a7dc30ddf67b"}},"options":{}}}
{"t":{"$date":"2023-11-24T01:40:22.825+01:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
{"t":{"$date":"2023-11-24T01:40:22.825+01:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"<IP_ADDRESS>"}}
{"t":{"$date":"2023-11-24T01:40:22.825+01:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
{"t":{"$date":"2023-11-24T01:40:22.830+01:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"LogicalSessionCacheRefresh","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"2ee0d9ec-bc3f-44b3-ac7e-a7dc30ddf67b"}},"namespace":"config.system.sessions","index":"id","ident":"index-5-7274828463766787221","collectionIdent":"collection-4-7274828463766787221","commitTimestamp":null}}
{"t":{"$date":"2023-11-24T01:40:22.830+01:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"LogicalSessionCacheRefresh","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"2ee0d9ec-bc3f-44b3-ac7e-a7dc30ddf67b"}},"namespace":"config.system.sessions","index":"lsidTTLIndex","ident":"index-6-7274828463766787221","collectionIdent":"collection-4-7274828463766787221","commitTimestamp":null}}
2023-11-24T00:40:25.487108Z  INFO text_generation_launcher: Peft model detected.
2023-11-24T00:40:25.487144Z  INFO text_generation_launcher: Loading the model it might take a while without feedback
2023-11-24T00:40:25.971497Z ERROR download: text_generation_launcher: Download encountered an error: The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling transformers.utils.move_cache().
0it [00:00, ?it/s]
0it [00:00, ?it/s]
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 261, in hf_raise_for_status
response.raise_for_status()
File "/opt/conda/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/transformers/utils/hub.py", line 429, in cached_file
resolved_file = hf_hub_download(
Error: DownloadError
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1195, in hf_hub_download
metadata = get_hf_file_metadata(
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1541, in get_hf_file_metadata
hf_raise_for_status(r)
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 277, in hf_raise_for_status
raise GatedRepoError(message, response) from e
huggingface_hub.utils._errors.GatedRepoError: 401 Client Error. (Request ID: Root=1-655ff0f9-2aa08e0f5f5956ce13367ba4;87818dcf-9985-4de5-87c2-f84407973c56)
Cannot access gated repo for url https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json.
Repo model meta-llama/Llama-2-7b-hf is gated. You must be authenticated to access it.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py", line 16, in download_and_unload_peft
model = AutoPeftModelForCausalLM.from_pretrained(
File "/opt/conda/lib/python3.10/site-packages/peft/auto.py", line 101, in from_pretrained
base_model = target_class.from_pretrained(base_model_path, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 494, in from_pretrained
resolved_config_file = cached_file(
File "/opt/conda/lib/python3.10/site-packages/transformers/utils/hub.py", line 444, in cached_file
raise EnvironmentError(
OSError: You are trying to access a gated repo.
Make sure to request access at https://huggingface.co/meta-llama/Llama-2-7b-hf and pass a token having permission to this repo either by logging in with huggingface-cli login or by passing token=<your_token>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/text-generation-server", line 8, in
sys.exit(app())
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 131, in download_weights
utils.download_and_unload_peft(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py", line 24, in download_and_unload_peft
model = AutoPeftModelForSeq2SeqLM.from_pretrained(
File "/opt/conda/lib/python3.10/site-packages/peft/auto.py", line 85, in from_pretrained
raise ValueError(
ValueError: Expected target PEFT class: PeftModelForCausalLM, but you have asked for: PeftModelForSeq2SeqLM make sure that you are loading the correct model for your task type.
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to <IP_ADDRESS> port 8080: Connection refused
Warning: Transient problem: connection refused Will retry in 10 seconds. 59
Warning: retries left.

Does your huggingface account have access to this repo: https://huggingface.co/meta-llama/Llama-2-7b-hf ?

I can see it, but then how do I get access to it?

did you merge adapter weights and remove adapter files?

how do you do that?

would this help, if i set the merger-adapter to true, see below


[FR-219] Allow normal messages to use masked links

can you still export the Markdown component for other use?

there are some places Discord doesn't render link markdown, such as user bios
not sure if we currently need to use it, but we should keep it in for future use


Add a configuration option to avoid consumption of magic links from security products

The purpose of this pull request is to add the ability to configure the magiclinksdev project to prevent robots, specifically email security products, from consuming magic links.
See this issue for more details and discussion: https://github.com/MicahParks/magiclinksdev/issues/2

These changes allow the project to be configured to block robots from consuming magic links. Here are some implementation details:

An HTML page will be returned. The page is self-contained except for the external reCAPTCHA v3 script.
On page load, once reCAPTCHA is "ready", an XMLHttpRequest POST request containing the reCAPTCHA token will be made to the same URL with a query parameter containing the token. If this passes on the backend, the target of the magic link will be returned the response body and redirected to with window.location.replace(xmlHttp.responseText);.
There is an additional configuration for a "button bypass", which includes an HTML <form> in the page. When that form is submitted via a button click, the page will be redirected via an HTTP redirect to the target of the magic link. This is a JavaScript-free alternative for redirecting user in the event that the user's web browser does not run JavaScript, or reCAPTCHA v3 validation fails. This "button bypass" feature is non-default, because it is plausible that a security product would click this button and submit the form, however that should be unlikely.

Here are the renderings of the HTML pages used for redirection. These pages should only show for a brief moment, then automatically redirect.
Default:

"button bypass":


ValueError: Unknown layer: Functional

Currently running this in Windows with conda (the version of tensorflow-gpu ended up being 2.1.0, latest supported package version with conda at this moment)
While following the instructions in the README I found myself with this error when trying to run S3_network_testing.py with the the prepared dataset and pretrained model:
Error: Traceback (most recent call last):   File "S3_network_testing.py", line 37, in <module>     model = tf.keras.models.load_model(model_path,custom_objects={'iou':fun.iou})   File "C:\Users\eriver08\Miniconda3\envs\XYZNet\lib\site-packages\tensorflow_core\python\keras\saving\save.py", line 146, in load_model     return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)   File "C:\Users\eriver08\Miniconda3\envs\XYZNet\lib\site-packages\tensorflow_core\python\keras\saving\hdf5_format.py", line 168, in load_model_from_hdf5     custom_objects=custom_objects)   File "C:\Users\eriver08\Miniconda3\envs\XYZNet\lib\site-packages\tensorflow_core\python\keras\saving\model_config.py", line 55, in model_from_config     return deserialize(config, custom_objects=custom_objects)   File "C:\Users\eriver08\Miniconda3\envs\XYZNet\lib\site-packages\tensorflow_core\python\keras\layers\serialization.py", line 106, in deserialize     printable_module_name='layer')   File "C:\Users\eriver08\Miniconda3\envs\XYZNet\lib\site-packages\tensorflow_core\python\keras\utils\generic_utils.py", line 292, in deserialize_keras_object     config, module_objects, custom_objects, printable_module_name)   File "C:\Users\eriver08\Miniconda3\envs\XYZNet\lib\site-packages\tensorflow_core\python\keras\utils\generic_utils.py", line 250, in class_and_config_for_serialized_keras_object     raise ValueError('Unknown ' + printable_module_name + ': ' + class_name) ValueError: Unknown layer: Functional
I managed to fix it by modifying lines in:
S3_network_testing.py

Line 37: wrap fun.iou in a list

model = tf.keras.models.load_model(model_path,custom_objects={'iou':[fun.iou]})
My guess is that it could be a tensorflow version issue within the conda environment (some version supports the function as it is and another one requires it to be in a list).
Also I noticed that there was another problem in the training script that wouldn't let me train the model from scratch:
S2_network_training.py

Line 83: wrap fun.iou in a list
model.compile(opti,loss ='categorical_crossentropy',metrics=[fun.iou])
Line 87: Comment this line (ShapeNet_model_16.h5 is a different version that was being trained locally?)
Line 88: Uncomment this line (actual name of the pretrained model is defined in saved_path)

I am guessing this last problem was just some code changes that were not supposed to be pushed to master.

fun.iou is located at fun_network.py Line19 - 43. It seems that you failed to load this customized from a saved model. I guess it is due to your windows version of Tensorflow.
I would suggest you to try on a Ubuntu version. An alternative solution would be dragging the iou function to S2_network_training.py, defining the network directly and try to load weight instead of loading the model.
Good luck!
Yecheng

Thanks but actually the "issue" I added was more of a documentation for people trying to implement this using the given steps in Windows since I did get it working (I mention that before referencing modifications) and it did work with the pre-trained model. Very nice work btw!

Hi Ykram12,
Thanks for trying our approach and letting us know about this issue! Please do let us know if there are more issues in the future. Thanks.


pytorch / pytorch-bin behave diffrently wrt cuda

Describe the bug
pytorch / pytorch-bin behave handle cuda differently
Steps To Reproduce
Steps to reproduce the behavior:
nix-shell -p python310Packages.pytorch --run 'python3 -c "import torch; print(torch.cuda.is_available())"'
False
nix-shell -p python310Packages.pytorch-bin --run 'python3 -c "import torch; print(torch.cuda.is_available())"'
True
Expected behavior
i would expect the same behavior
Ideally both would return True if cuda is supported
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[lizelive@l05:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.63, NixOS, 22.05 (Quokka), 22.05.2785.cc634d9aa08`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

(for tensorflow they both don't have cuda)

Not sure there's much to be done here. The binary Pytorch wheels ship with the redistributable CUDA binaries because that's how upstream has chosen to distribute them them, and evidently Tensorflow wheels don't come with them. Nixpkgs does not enable CUDA support in source builds by default, both because CUDA is unfree software so Hydra will not build them anyway, and because CUDA toolkits cannot be redistributed with the source components.
You are welcome to build Pytorch from source with CUDA yourself simply by overriding with cudaSupport = true; but be warned this will take a few dozen CPU hours; less if you set the cudaArchList to only what you need for your specific card. Alternately, you can use the Nixpkgs CUDA maintainers cache as described on the wiki, or just use the binary distribution of Pytorch.

For future googlers, check out the python3Packages.torchWithCuda package which should do what you want. However, cache.nixos.org doesn't keep binary caches for that package since it includes CUDA which is unfree, but if you use the cuda-maintainers cache you should be able to avoid having to build from source. You can find more info in the wiki here.
I'm going to close this for now, since I think this is basically working as intended (or at least as closely as possible) as @tpwrules notes.


Remnux fresh install failed on focal

renmux install fails at the same point after a couple of tries, specifically after SSH. Here are the result: failed parts of the log:
Update: I deleted everything and tried again. Here are the latest fails.
start_time: '18:33:24.410852'

file_|-remnux-addon-version-file_|-/etc/remnux-version_|-managed:
run_num: 757
sls: remnux.addon
changes: {}
comment: 'One or more requisite failed: remnux.packages.remnux-packages'
duration: 0.012
result: false
start_time: '18:33:22.529314'

pkg_|-remnux-packages-rar_|-rar_|-installed:
id: remnux-packages-rar
run_num: 221
sls: remnux.packages.rar
changes: {}
comment: "Problem encountered installing package(s). Additional info follows
:\n\nerrors:\n
\   - Running scope as unit: run-rec59e57475fd477d9e5274bd3a909c54.scope\n
E:
Package 'rar' has no installation candidate"
duration: 1304.853
name: rar
result: false
start_time: '18:41:37.259814'

test_|-remnux-dedicated_|-remnux-dedicated_|-nop:
run_num: 814
sls: remnux.dedicated
changes: {}
comment: 'One or more requisite failed: remnux.addon.remnux-addon-version-fi
le'
duration: 0.009
result: false
start_time: '18:42:21.775217'

test_|-remnux-packages_|-remnux-packages_|-nop:
run_num: 225
sls: remnux.packages
changes: {}
comment: 'One or more requisite failed: remnux.packages.rar.remnux-packages-
rar'
duration: 0.013
result: false

Fixed. RAR wouldn't install as I didn't have multiverse repo enabled.

Hi @TheWanderer1983 , from looking at the logs, it appears that you ran out of space in the VM and some of the tools didn't install:
ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device"
You may have to increase your disk space or clear some up room and run the installer again.


remove parallel assembler as no longer needed

Description of the PR
remove parallel assembler as no longer needed
PR Checklist

[ ] All commits have a Developer Certificate of Origin (DCO) -- they are generated using -s flag to git commit.
[ ] All new changes are covered by tests
[ ] If GraphQL schema is changed, make generate has been run
[ ] If collectsub protobuf has been changed, make proto has been run
[ ] All CI checks are passing (tests and formatting)
[ ] All dependent PRs have already been merged

Recreated this PR as the other one auto-closed after rebasing..


Handle routing state and replace

A couple of different changes here. Could split into multiple PRs, but wanted to see if you're interested first (I've also of course seen the PRs that already exist, but just copied this from the changes I needed for my app)
The changes:

No longer use window.location, always rely on the location we receive from history
Because of the above, let the initial state not be the current url, but set is with noRouterUpdate=true
Add handling of location.state (and therefore a deep equal check when comparing locations)
Use history.createPath instead of locationToString
Make it possible to replace in addition to push

Also thought about renaming, but didn't for this PR:

updatePath to pushPath (closer to history.push and history.replace)
noRouterUpdate to skipUpdate or noUpdate
syncReduxAndRouter to syncReduxAndHistory

This is looking good. I really don't want to do a deep comparison on state though; luckily, I figured out they comparing locations is the wrong way to detect changes anyway. See the new sync logic at https://github.com/jlongster/redux-simple-router/pull/40. I'm going to merge it after I write some basic tests.
I need this to be rebased on top of that once I land that. Thanks a lot for this PR! Either go ahead and rebase or I'll ping you when it's merged.

Also see https://github.com/jlongster/redux-simple-router/issues/21#issuecomment-156637886. I'm not sure if we want to change actions names or not.

Since this is already changing the API for updatePath, any thoughts on losing the ordinal boolean parameters? updatePath( path, null, true ) doesn't read as well as updatePath( path, null, { noRouterUpdate: true } )
Or for that matter, since state will be optional most of the time the signature could be function updatePath( path, { state, noRouterUpdate } = {} )

I'm down with that. Let's go ahead and change the API to pushPath and replacePath. Does that sound good? Or "route" instead of "path".

Matching the naming scheme used by history makes the most sense to me, but it sounds like they might be renaming things as well (to just push and replace) so who knows.

Yeah, that's what Ryan said. Also I don't think pushState really says anything about routing outside of react-router, so seeing that action in the wild isn't the best name IMHO, but push and replace are even more vague. I don't think there's anything wrong adding more semantics into the name, and they obviously map to push/replace.

I just landed the new sync logic. Can you rebase?

And also add tests!

@jlongster Great — I'll rebase and fix the tests asap

I think this PR will reintroduce the problem with devtools where a 'RESET' action will change the path to '/' regardless of the initial URL. In my opinion the benefits of landing this outweigh this devtools regression, but it would be nice if we could figure out a way round it.

This is the issue where it was originally "fixed" (modulo my typo of not quoting undefined) https://github.com/jlongster/redux-simple-router/pull/6

@ellbee that's true, I didn't notice we lost that check. Should be easy to add back in the check to see if we have a window. However, I've been thinking that getting the URL from window.location isn't the best either. Because if you are using hash history you just want to get /foo, not /#/foo?_key=blah.
Maybe what we should do is set initialState to null/false/etc, but in the initial history.listen call (the listener is always called immediately when you add a listener) we do initialState = { path: locationToString(location), ... } and change the initialState. We wouldn't be mutating the object, we would just define it with let so we could override it later.

Although I guess we'd have to keep track if we've set the initialState or not and not do it multiple times... I don't love it, but history does not have a way to get the "current location".

@jlongster Rebased and added tests. I also fixed a bug with avoidRouterUpdate. Let me know what you think.
(I thought about cleaning up a couple of things in the tests, but I'll do that in a follow-up PR — this is already a big change)

Thanks a bunch for this, it's awesome! I'll try to merge it soon, just wanted to look through it once more. It is thanksgiving also :)

@jlongster Thanks for the comments — I'll take a look and try to get it fixed without the deep-equal :)

What was the bug with avoidRouterUpdate btw?

@jlongster The bug: https://github.com/kjbekkelund/redux-simple-router/commit/a0d6945f94f3ca1ec45d180f43a8c8ccbbab4fac
It's specified as updatePath(path, avoidRouterUpdate), but used as an object updatePath(routePath, { avoidRouterUpdate: true }).
Btw, pushed a test that would fail if we remove the locationsAreEqual check without other changes (too many store updates plus it changes replace -> push in the store). I'll look at ways of removing locationsAreEqual

Although I guess we'd have to keep track if we've set the initialState or not and not do it multiple times... I don't love it, but history does not have a way to get the "current location".

Ok, thanks. That sounds like a good plan.

Thanks all. I have to leave and it's Thanksgiving so I'm going to be slow to reply over the next day or two. I'm happy to merge this in with deep-equal for now and then we can look at it later as well. @kjbekkelund is this ready to merge if we just go with deep-equal for now?

@jlongster Everything looks ok when testing in my app, so it should be ready to merge

I found a way to remove deep-equal. I think we need to be sure it's the right way to go, so it should be its own PR (e.g. to discuss additional tests etc). I'll create the PR against master when this is merged.

Cool! I think I'm going to wait until tomorrow night to merge. We got a lot of attention today and I just don't want to merge something big right before a day when I'm not going to be around, even though it's just the master branch.

👍 Relax and enjoy Thanksgiving!
Got some ideas to solve deep-equal, but I think we need tests running in browsers to ensure everything still works as expected.

Alright let's just merge this now and clean it up over the next few days for a release.

Thanks a lot for all the work @kjbekkelund and others!


release: update core docker image

update core docker image for go monorepo to compile against.

Should this include a new quickstart image release including https://github.com/stellar/go/issues/4723?


Please provide your ORCID

Hi !
in order to publish your research in the Sci-GaIA Open Access Repository, you need to have an ORCID. Please respond to this ticket with your ORCID : e.g. orcid.org/0000-0002-6607-7145
If you do not have one, please sign up at https://orcid.org/register

My orcid ID: orcid.org/0000-0003-4448-7829
Thanks.
On Wednesday, November 23, 2016 2:33 PM, Bruce Becker<EMAIL_ADDRESS>wrote:

Assigned #32 to @damilare88.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks very much. Please take a few minutes to update the information (institute, etc.)


How to use fasterRcnn

The fasterRcnn model downloaded from your description is just a weight. How can this weight be added to the model in Encoder? How do I write this model?
Thank you!

Please read the Readme and related processing code carefully.

ok,thank you!


WIP: Fix the redis backend

Codecov Report

Merging #1 into master will increase coverage by 0.27%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master       #1      +/-   ##
==========================================
+ Coverage   59.79%   60.06%   +0.27%
==========================================
  Files          78       78
  Lines        6143     6143
  Branches      754      809      +55
==========================================
+ Hits         3673     3690      +17
+ Misses       2292     2279      -13
+ Partials      178      174       -4

Impacted Files
Coverage Δ

frontera/utils/tester.py
26.16% <0%> (ø)
:arrow_up:

frontera/contrib/backends/memory/__init__.py
69.67% <0%> (+0.81%)
:arrow_up:

frontera/_version.py
31.61% <0%> (+1.47%)
:arrow_up:

...rontera/contrib/backends/redis_backend/__init__.py
80.48% <0%> (+1.82%)
:arrow_up:

frontera/contrib/backends/hbase/domaincache.py
73.09% <0%> (+2.69%)
:arrow_up:

frontera/utils/msgpack.py
66.66% <0%> (+11.11%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8830ce2...2160c69. Read the comment docs.


Should reference model initialize weights uniformly?

Thanks for your awesome work! I noticed that the weights of the baseline model for Redpajama are not initialized uniformly. Does this mean that the reference model can be initialized with any settings?
https://github.com/sangmichaelxie/doremi/blob/main/configs/rp_baseline.json

Thank you for your prompt reply! I have one more question. I'm using my own dataset, which is also RedPajama but organized differently. Even after the third round of iterated DoReMi, I've noticed that the domain weights still haven't converged. What might affect the number of rounds I need?
P.S. I'm using the same training settings as in run_rp_doremi280M.sh, except I had to lower the per_device_train_batch_size to 20 due to hardware limitations. Following is my result of each round.

Round 1
Round 2
Round 3

common_crawl
0.661
0.580
0.592

c4
0.187
0.162
0.186

github
0.024
0.052
0.040

wikipedia
0.062
0.088
0.086

book
0.037
0.036
0.042

arxiv
0.013
0.040
0.025

stackexchange
0.017
0.043
0.028

Thank you so much for your reply.

Thank you for your prompt reply! I have one more question. I'm using my
own dataset, which is also RedPajama but organized differently. Even after
the third round of iterated DoReMi, I've noticed that the domain weights
still haven't converged. What might affect the number of rounds I need?

In my experience with running RedPajama (albeit with an older version of
the code), the weights for common crawl should typically go down while c4
goes up (even if common crawl is very high in the beginning). If you do
iterated DoReMi from uniform weights (for 2 rounds) I've seen that c4
becomes the dominant domain. You might find that you need more rounds
particularly if the initial reference domain weights are far from the
converged value.
On Wed, Sep 6, 2023 at 1:01 AM ChloeAuYeung @.***>
wrote:

Thank you for your prompt reply! I have one more question. I'm using my
own dataset, which is also RedPajama but organized differently. Even after
the third round of iterated DoReMi, I've noticed that the domain weights
still haven't converged. What might affect the number of rounds I need?
P.S. I'm using the same training settings as in run_rp_doremi280M.sh,
except I had to lower the per_device_train_batch_size to 20 due to
hardware limitations. Following is my result of each round.
Round 1 Round 2 Round 3
common_crawl 0.661 0.580 0.592
c4 0.187 0.162 0.186
github 0.024 0.052 0.040
wikipedia 0.062 0.088 0.086
book 0.037 0.036 0.042
arxiv 0.013 0.040 0.025
stackexchange 0.017 0.043 0.028
—
Reply to this email directly, view it on GitHub
https://github.com/sangmichaelxie/doremi/issues/11#issuecomment-1707860697,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB4J2SVBYSCSODNDMSXUN3DXZAUXLANCNFSM6AAAAAA4LJMSBY
.
You are receiving this because you commented.Message ID:
@.***>


Cross-compiling (ungoogled) Chromium for Windows from Linux

...without touching a Windows PC.
This should make possible a u-c build that is...

More easily reproducible and CI-friendly, since no manual "install Visual Studio on your PC" step is involved;

More hermetic, in that we don't have to worry about said PC having been surreptitiously compromised;

Faster, in line with the experience of others;

Less costly to run, given how GitHub double-counts each minute of GitHub Actions that runs on Windows as compared to Linux.

Google has posted official instructions on how to cross-compile Chromium for Windows, but that documentation makes some CI-averse assumptions (packaging up a manual Visual Studio installation) and glosses over several important details (e.g. SetEnv files and the rc executable).
Even more unfortunately, I could not find that anyone has distilled those instructions into a working setup that others can use as a starting point. So, given that I've already built this beast (for Linux) more times than I care to count, I figured I'd take a stab at it.
To begin with, I'd like to draw your attention to a nifty little project called msvc-wine. They have put together a Python script (vsdownload.py) that can download any of Microsoft's current SDK offerings, without needing to run some setup/installer/wizard thing. This eliminates the requirement of a Windows PC altogether.
For compiling Windows resource files, Google has a re-implementation of rc that they use in the cross build. The source is available here. (That may seem like a random location, but it is directly referenced in the Chromium tree. Nico appears to be a Googler.)
Many guides for performing cross builds have some variation of a "put the Windows SDK on a VFAT-formatted filesystem" step, to recreate the case-insensitive lossage of the target OS. (You have winsock2.h vs. WinSock2.h, advapi32.lib vs. AdvAPI32.Lib etc., and files are often referenced via both forms.) This was more hassle than I wanted to deal with, so I wrote up a script (case-fold.sh) that creates a bunch of symlinks to get the same effect.
And with all that, I've put together tooling to build a Docker image that provides a suitable Linux environment (based on Debian unstable or Ubuntu mantic) for performing the cross build. I've already run through building 119.0.6045.159 with it several times.
Unfortunately, the build is not fully working yet---with -k 0, it gets a bit past the halfway mark before stopping. While all the tooling issues I've come across have been addressed, there are a number of funky C++ errors that I can't make heads or tails of... things like implicit StringPiece conversions not working correctly. I'm using Clang 17, so I don't think it's an issue of the compiler not being new enough. I'm hoping that folks who have more experience with this codebase, and with the Windows side of things in general, will be able to make sense of what's going on.
Here is a log file produced by a second invocation of ninja ... -k 0 chrome that shows all the error messages, strung together. If you are successful in reproducing my build, then this is what you should see.
Steps to get this working:

git clone the relevant source location onto a system with a Docker host;

Edit the top of the Dockerfile to choose Debian or Ubuntu as your base (it should not take much work to adapt the image build to some other distro, like Arch, but this is what I can provide), and edit APT_MIRROR to point to a faster/local APT host if you have one;

Edit the setup.sh script to uncomment the accept_license variable, by which you accept the Microsoft EULA mumblemumble;

Build the Docker image with make build-image;

If that is successful, then you can start a container with make run;

Unpack a Chromium source tarball inside the container;

In the top-level directory of the Chromium source tree, run the ~/prepare-source.sh script. This will fix a few minor source issues, put some necessary files into place, and run gn gen with a configuration that is known to work;

Run e.g. ninja -C out/Release -k 0 chrome, and see how far it gets;

To delete the image, exit the container and run make clean.

Of course, you will have to customize the container workflow to suit your needs (add mounts, change the user ID, etc.). Some points to keep in mind:

All the Microsoft SDK stuff is installed under /opt/microsoft/, and the Chromium-relevant environment variables are set accordingly;

Google's rc is available in source form under /usr/local/src/, and binary under /usr/local/bin/;

All the necessary compiler and library (distro) packages that I'm aware of are installed. More may be needed by later stages of the build;

Additional case-variation symlinks under /opt/microsoft/ may be needed by later stages of the build. These will have to be added to the case-fold.sh script;

If you build the image more than once, then I suggest tarring up the files under ~/cache/, copying that out, and then uncommenting the ADD ... cache.tar ... directive in the Dockerfile. This will allow you to run through the MSVC "install" process again without re-downloading the ~2.5 GB lot from Microsoft;

The scripts are reasonably commented to explain what's going on, so please feel free to read through them beforehand;

I am more than happy to reproduce and sort out tooling issues as you encounter them. What I need help with is the Chromium codebase itself, as the C++-fu in use is beyond my ken;

As you can see, I haven't even bothered to bring in the u-c patches or tooling yet. Gotta walk (i.e. build plain Chromium) before you can run, I say! But that is naturally the end goal.

Some additional notes:
Google has a CI instance covering this cross build, but it has been broken for months. The most recent successful build that I could find (without spending too much time looking for it, given the poor UI) dates from the end of May.
Here is the GN config they use, though this doesn't work for our purposes (e.g. we need use_sysroot=false since we're not using a Linux sysroot):
dcheck_always_on = true
is_clang = true
is_component_build = true
is_debug = false
llvm_force_head_revision = true
symbol_level = 1
target_os = "win"

Hi @uazo!

They have put together a Python script (vsdownload.py) that can download any of Microsoft's current SDK offerings,

interesting, thanks for finding it! Do you mind if I take a cue from it?

Go right ahead! The msvc-wine project figured out a solution to a rather tricky problem, and we should all take advantage of it.
Just be sure to include appropriate caching logic, so that the MSVC files are not downloaded from Microsoft more than necessary. The vsdownload.py script has a --cache option, similar to other package-downloading tools, that helps with this.

The source is available here.

correct, at the beginning (last year) I would download it and build it (https://github.com/uazo/bromite-buildtools/blob/master/images/chr-source/prepare-build.sh#L77) but then I saw that it is already available (pre-compiled by google https://github.com/uazo/cromite/blob/master/tools/images/bromite-build/pre-start.sh#L70)

It's better to avoid non-distro binaries, just on principle. While I doubt Google would provide a bad binary, if (for example) you ever want to run the build on a different platform, then you'll have to reach for the source code anyway.

Many guides for performing cross builds have some variation of a "put the Windows SDK on a VFAT-formatted filesystem" step, to recreate the case-insensitive lossage of the target OS.

This is what I do, I use a case-fold disk (the runner is on my server) but your solution is intriguing

There's no need for the full generality of a case-folding filesystem, because SDK files are not accessed via arbitrary case permutations---in all instances I've seen, the build needs the lowercase name, a canonical mixed-case (or all-uppercase) name, and that's it. So symlinks are enough for that.

you can find my solution in my repo if you want to compare it, I use docker in docker and sysbox for the build but I think it is also applicable in the github runner if disk space permits. the build has been working for years without any problems, you can also find the images in docker hub

Thank you, this is very helpful!
I tried building with your patches and GN args, and... I'm happy to say... I got a successful build :smiley:
I had to make some updates to case-fold.sh and a couple other scripts, which I've already pushed to my branch. And now I have to figure out the salient differences between your configuration which worked, and mine that didn't. But this unblocks me, so I greatly appreciate that!
Now, looking at your setup... I see you're using depot_tools, gclient, goma, the casefold location... there are a lot of moving parts. I set out from the beginning wanting to avoid all that kind of custom tooling; I wanted a build that was similar in "shape" to building a package for a Linux distro.
One benefit of having the Google tooling in working order, however, is being able to identify exactly what Microsoft SDK version they're using. The vsdownload.py script lets you specify a version, but otherwise it downloads the latest of what's available from Microsoft. Between last week and now, I noticed that the latest SDK version got updated (more specifically, I saw that the script downloaded some files in addition to using the --cache directory I provided, whereas before, it didn't download anything). We would be better off matching the SDK version we use with whatever Google's using, to avoid any build issues due to minor incompatibilities between the two. But we would need a (reliable, automatic) way of telling what version Google is using.

Just be sure to include appropriate caching logic, so that the MSVC files are not downloaded from Microsoft more than necessary.

thanks but i don't think it's necessary for my purposes. after all, it's not possible to distribute the result (the ms licence doesn't allow it) so it'll probably be something i'd launch with every change of major (once a month).

So symlinks are enough for that.

In fact, I like your solution. the only problem is that it must be maintained.

Thank you, this is very helpful!

I am glad.

I see you're using depot_tools, gclient, goma, the casefold location... there are a lot of moving parts

yes, some things I don't use anymore and I could get rid of them, but it's just a matter of priorities and unfortunately doing everything myself is quite a challenge. if you have any doubts, just ask!

is being able to identify exactly what Microsoft SDK version they're using.

sure, here it is
https://source.chromium.org/chromium/chromium/src/+/main:build/vs_toolchain.py;l=43;drc=8e78783dc1f7007bad46d657c9f332614e240fd8?q=build%2Fvs_toolchain.py&ss=chromium%2Fchromium%2Fsrc

thanks but i don't think it's necessary for my purposes. after all, it's not possible to distribute the result (the ms licence doesn't allow it) so it'll probably be something i'd launch with every change of major (once a month).

This would go e.g. in a runner's non-public storage area. Of course the files can't be redistributed, but that doesn't rule out retaining them for a future run.

So symlinks are enough for that.

In fact, I like your solution. the only problem is that it must be maintained.

At worst, it would be sporadic breakage due to some new MS header being pulled in via a mixed-case name. I would not expect that to happen on the regular.
Alternately, one could scan the MS headers and the Chromium source for new mixed-case names, and create those symlinks ahead of time. But that could end up being a lot of work up front to avoid a relatively small amount of work in the future...

I see you're using depot_tools, gclient, goma, the casefold location... there are a lot of moving parts

yes, some things I don't use anymore and I could get rid of them, but it's just a matter of priorities and unfortunately doing everything myself is quite a challenge. if you have any doubts, just ask!

Will do, thank you :smile:  I hope my work can make the task easier to deal with.

is being able to identify exactly what Microsoft SDK version they're using.

sure, here it is https://source.chromium.org/chromium/chromium/src/+/main:build/vs_toolchain.py;l=43;drc=8e78783dc1f7007bad46d657c9f332614e240fd8?q=build%2Fvs_toolchain.py&ss=chromium%2Fchromium%2Fsrc

Ah, so that's where it's controlled. That covers the SDK version... now, there is also a VC tools version in the mix (i.e. $WINDOWSSDKDIR/VC/Tools/MSVC/14.38.33130/). The vs_toolchain.py script has that MSVC_TOOLSET_VERSION table, but it only yields VC143, not 14.38.33130. Would you know offhand how the specific tools version is obtained? I'm not seeing anything obvious in the script.

VC143, not 14.38.33130
Would you know offhand how the specific tools version is obtained?

no, sorry, I don't know.

All right, I've pushed a major update to my branch. Here are the highlights:

The prepare-source.sh script is gone, replaced by new patches under patches/general/ and a provided args.gn file. If you are building the upstream Chromium tarball, then this is what you'd use;

Otherwise, I've started integrating this into the u-c build. New patches have been added to the patch series, and a couple of existing ones tweaked. NOTE: The non-cross Windows build should not be affected, as any different behavior is conditionalized on a Linux host platform;

There is now a modified build.py script under cross-build/ to drive the build. Eventually, this should get merged into the top-level build script (mostly it's just conditionalizing out the Windows-isms);

Re-generating IDL source files is needed in the u-c build (as the pre-generated files get pruned), and because this can only be done with Microsoft's MIDL, I've integrated the use of Wine into the Docker image. You can see the setup for this in the new windows-run-midl-via-wine.patch. Thankfully, this is all that Wine is needed for.

I've confirmed successful cross builds for Chromium 120.0.6099.109, with both the (lightly patched) upstream source, and the u-c patched source.

I've pushed a major update to my branch.

Uhm... sorry, where is it? can it be seen?

I've pushed a major update to my branch.

Uhm... sorry, where is it? can it be seen?

Of course; everything is here: https://github.com/iskunk/ungoogled-chromium-windows/tree/feature/win_cross

very complex for my skills (few to tell the truth), congratulations. I think I will try the part of the prerequisites and the casefold :) I will have to study it. I will let you know.

The case-fold.sh script should now be complete, in terms of the symlinks it creates. It'll also cover x86 as well as x64 (though x86 cross builds are currently broken for other reasons).
I haven't had time to write up a proper README on this, unfortunately---I pushed this out now to get ahead of the holiday rush---but if you run into any difficulties, please feel free to ask about it. (Likewise for anyone else following this!)

@iskunk Would it be possible to document your findings as a readme for this repository?

Hi @PF4Public! Another big update is in. I've added a README file with the new build process, not least because the one described at the top of this issue is obsolete. I saw Rust is now required for the build, and that has been covered as well.
The whole build is working for me. I get what are presumably working installers with presumably working browsers at the end of the run. But I have no means of testing them, because any Windows box over which I have a modicum of authority tends to become a Linux box ^_^  So at this point, I am really needing others to try out this build process, and especially to verify that the end result actually works.
One significant change: I was originally planning to augment the top-level build.py script so that it can drive the cross build as well as the regular one. But eventually, I figured... there's so much heavy lifting that script has to do on Windows, that is just not applicable in the Linux environment. This was basically going to end up spamming if arch == 'win32': everywhere. It was clear that I could write a much shorter and simpler build script in shell, hence the new build.sh file. As a bonus, it even supports re-starting a previously-interrupted build (implemented for my own sanity's sake, because unpacking that ginormous tarball takes a long time even on the monster systems I'm testing this on).
Again, I really need feedback on this. The build process itself, as far as I can tell, is feature-complete at this point. It's now about making this easy/robust for others to use, documenting it where needed, and of course, making sure that the resulting binary doesn't immediately go splat. There's not a whole lot further I can take this by myself.
(Note: I've just confirmed that x86 build support is currently broken, due to a missing 32-bit header file. Will be looking into that.)

I've updated the branch, and the x86 build is working now. (I doubt it was working previously.) It needed a few additional packages in the build container, and oddly enough, a new patch.

Do you plan to submit a PR eventually?

Of course. I could first use some independent validation of what's there, however. It's not a good look to say, "I don't know if the resulting build actually runs, 'cause no one's tried it." As well as the squishier points of approachability and ease of use---I'm too close to the work to evaluate this myself.

I'm too close to the work to evaluate this myself.

would you allow me?
you should investigate the use of ciopfs by the chromium toolchain, I think it would be the ultimate solution, whether it works.

Hi again @uazo,
By all means, feel free to try this out, and let me know if you run into any hard spots. If you are working on Windows, you may want to try using Rancher Desktop for the container runtime. Eventually, I would like that to be the path for non-technical users on Windows who want to run this build.
I get what ciopfs is doing, but anything involving mounts is dicey inside a container. Symlinks are a lot simpler to deal with. The only advantage that a case-insensitive filesystem has over that is that it will handle any new accessed files that come up in the future, whereas symlinks have the potential of breaking. But this does not appear to be a common occurrence---when I re-ran this build, after months of not touching it, there was no breakage due to missing symlinks.

Hi @uazo, have you had a chance to try out the cross build?
I've been on a tear lately, pushing dozens of branch updates to get this up and running as a GitHub Workflow, and making numerous fixes/adjustments as a result. Still no test of the final binary, however.

have you had a chance to try out the cross build?

I'm sorry, I don't understand. you mean in your mode? I already do a windows cross build all the time.

I'm sorry, I don't understand. you mean in your mode? I already do a windows cross build all the time.

I should have been clearer; yes. I could use some independent confirmation that the process I've put together is usable by more than just me, and hopefully feedback on any issues and potential improvements. A big part of this work is making the cross build more accessible, so that it can be run by the same enthusiast-but-not-developer users who currently compile Chromium/ungoogled-chromium themselves.
The framework remains a work in progress, however. On my end, the issue is no longer about getting the cross build working, but about making it work reproducibly. So I hope you're up for a few rounds...

sorry, do not take offence, but I am not interested to test your solution because I do not consider it reliable. i am interested in your work with the download of microsoft sdk, i will try that.
try to understand me, I am focused on development, and every toolchain-related problem for me is work that annoys me and wastes my time.
I think you should ask the maintainer of this repo if he is interested in your work for testing, if he finds benefit in it.

I'm only asking for testing and feedback, not for this to replace any aspect of your current development process.This work occupies a very narrow technical niche that relatively few developers even approach, so your feedback would be quite valuable. Toolchain work is not glamorous, but the payoff of a robust and reliable end product is worth it, I find.
As for the maintainer of this repo, I'm sure they would have commented already if they were interested.

Thank you @iskunk!
I confirm that your windows sdk generation tool works perfectly.
Here is my version.
Just one thing; the line is incorrect: pushd %~dp0..\..\..\n --> pushd %~dp0..\\..\\..\n

Great to hear, @uazo! The msvc-wine/vsdownload.py script is the centerpiece of a much nicer way of dealing with Windows builds.
Good catch on the missing backslashes. I've incorporated that fix, and inspired by your post, have been working to update my cross-build tooling to current versions. It's time to start preparing all this for a PR.
Note that as of three days ago, msvc-wine now produces a Windows Kits directory instead of kits, since Clang et al. expect the former.


Runtime-safe array subscripting

I noticed you switched the Setting values and Getting values comments around.
Also, out of curiosity, why not simply do something like this:
var array = [0, 1, 2]

// Getting values
array.safeAccess[0] // same as array[0]
array.safeAccess[3] // out of bounds, evaluates to nil

// Setting values
array.safeAccess[0] = 42 // same as array[0] = 42
array.safeAccess[3] = 42 // out of bounds, no operation

Where safeAccess could be implemented like so:
public struct SafeAccessor<T: CollectionType> {
    let value: T

    public subscript(index: Int) -> Int? {
        get {
            return try? self.value[index]
        }
        set (newValue) {
            try? self.value[index] = newValue
        }
    }
}

public extension CollectionType {
    public func safeAccess() -> SafeAccessor<Self> {
        return SafeAccessor<Self>(self)
    }
}

It's all pseudocode, but essentially, it can be implemented using a reflected accessor, and doesn't need to be language level. I'll try to flesh this out later.

The try? self.value[index] syntax is not implementable for all collections without an undue burden that will affect performance in all cases (including collection[index]).  It is not always possible to detect an invalid index reliably.

@gribozavr Here's what you want:
public extension CollectionType {
	public subscript (safe index: Index) -> Self._Element? {
		get {
			return self.indices.contains(index) ? self[index] : nil
		}
	}
}

public extension MutableCollectionType {
	public subscript (safe index: Index) -> Self._Element? {
		get {
			return self.indices.contains(index) ? self[index] : nil
		}
		set(newValue) {
			if let val = newValue where self.indices.contains(index) {
				self[index] = val
			}
		}
	}
}

I've tested that it works exactly according to your spec.
var array = [0, 1, 2]
array[safe: 0] // same as array[0]
array[safe: 3] // out of bounds, evaluates to nil
array[safe: 0] = 42 // same as array[0] = 42
array[safe: 3] = 42 // out of bounds, no operation

That would cause the safe subscript to be O(n), which is unacceptable.

self.indices.contains() is the only new call here; I don't see where the documentation states its overhead.

In fact, it looks like the implementation is right here:
/// O(1) implementation of `contains()` for ranges of comparable elements.
extension Range where Element : Comparable {
  @warn_unused_result
  public func _customContainsEquatableElement(element: Element) -> Bool? {
    return element >= self.startIndex && element < self.endIndex
  }

  // FIXME: copied from SequenceAlgorithms as a workaround for
  // https://bugs.swift.org/browse/SR-435
  @warn_unused_result
  public func contains(element: Element) -> Bool {
    if let result = _customContainsEquatableElement(element) {
      return result
    }

    for e in self {
      if e == element {
        return true
      }
    }
    return false
  }
}

I'm taking this to mean _customContainsEquatableElement is O(1). It doesn't seem like the safe subscript is O(n) in this case.

Not all indices are comparable, and, that overload of contains() does not work from a generic context.

@gribozavr Are there cases you see O(n) with this safe subscript implementation? I think for the majority of cases, it would be O(1). Of course, I'm thinking of Sets, Arrays, and Dictionarys.

Even Array would be O(n) since, as I said, that overload of contains() can't be called from a generic context.
Also, for invalid indices, the comparison operation is allowed to return whatever (or even trap!), and the actual subscript operation might trap as well.

@gribozavr I see what you mean. I'll think about this a little more.

I am inclined to reject this PR because no use-cases are shown or discussed, and it doesn’t say what problem it’s solving.

This proposal has insufficient detail: it does not provide the declarations of the subscript operator to be added, provide motivating use cases, or explore the alternative at all. I suggest revising the proposal and posting it to swift-evolution again as a "[Proposal draft]" to see if it draws more attention with the additional detail.


Nohup: ignoring input (ERROR: Cannot find config file)

Hello,
I have just upgraded my gekko (git pull && npm update) and it started producing the following error:
cat history.log
nohup: ignoring input

    ______   ________  __    __  __    __   ______
   /      \ /        |/  |  /  |/  |  /  | /      \
  /$$$$$$  |$$$$$$$$/ $$ | /$$/ $$ | /$$/ /$$$$$$  |
  $$ | _$$/ $$ |__    $$ |/$$/  $$ |/$$/  $$ |  $$ |
  $$ |/    |$$    |   $$  $$<   $$  $$<   $$ |  $$ |
  $$ |$$$$ |$$$$$/    $$$$$  \  $$$$$  \  $$ |  $$ |
  $$ \__$$ |$$ |_____ $$ |$$  \ $$ |$$  \ $$ \__$$ |
  $$    $$/ $$       |$$ | $$  |$$ | $$  |$$    $$/
   $$$$$$/  $$$$$$$$/ $$/   $$/ $$/   $$/  $$$$$$/

	Gekko v0.4.6
	I'm gonna make you rich, Bud Fox.

 ERROR: Cannot find the specified config file.

nohup: 8.21
node: 6.10.1
npm: 4.2.0
Starting gekko with
nohup node gekko --config myconfig.js &> history.log &
I have been starting gekko via the same command and same config file for quite some time now. Suddenly it does not work.

Strange, it is working for me. What OS are you on?

Ubuntu 14.04.05
Also strange, I had to sudo npm update the last time, it was asking for administrator permissions for some update. I don't think I had to run that as root before.
I will try to collect some more info and see if I can track down the problem.

Can you double check if the file myconfig.js actually exists within the root folder of gekko? For me it is working fine on Ubuntu:
mike@main:~/projects/gekko$ node gekko --config myconfig.js

    ______   ________  __    __  __    __   ______
   /      \ /        |/  |  /  |/  |  /  | /      \
  /$$$$$$  |$$$$$$$$/ $$ | /$$/ $$ | /$$/ /$$$$$$  |
  $$ | _$$/ $$ |__    $$ |/$$/  $$ |/$$/  $$ |  $$ |
  $$ |/    |$$    |   $$  $$<   $$  $$<   $$ |  $$ |
  $$ |$$$$ |$$$$$/    $$$$$  \  $$$$$  \  $$ |  $$ |
  $$ \__$$ |$$ |_____ $$ |$$  \ $$ |$$  \ $$ \__$$ |
  $$    $$/ $$       |$$ | $$  |$$ | $$  |$$    $$/
   $$$$$$/  $$$$$$$$/ $$/   $$/ $$/   $$/  $$$$$$/

        Gekko v0.4.6
        I'm gonna make you rich, Bud Fox.

2017-04-02 11:44:20 (INFO):     Setting up Gekko in realtime mode
2017-04-02 11:44:20 (INFO):
2017-04-02 11:44:20 (INFO):     Setting up:
2017-04-02 11:44:20 (INFO):              Candle writer
2017-04-02 11:44:20 (INFO):              Store candles in a database
2017-04-02 11:44:20 (INFO):

2017-04-02 11:44:20 (INFO):     Starting to watch the market: Bitstamp BTC/USD
2017-04-02 11:44:20 (DEBUG):    scheduling ticks
2017-04-02 11:44:20 (DEBUG):    Requested BTC/USD trade data from Bitstamp ...

I normally give the full path to the config, but it is actually in the gekko directory.
Executing gekko from within the directory, gives,
node gekko --config BTCe-MACD.js

    ______   ________  __    __  __    __   ______
   /      \ /        |/  |  /  |/  |  /  | /      \
  /$$$$$$  |$$$$$$$$/ $$ | /$$/ $$ | /$$/ /$$$$$$  |
  $$ | _$$/ $$ |__    $$ |/$$/  $$ |/$$/  $$ |  $$ |
  $$ |/    |$$    |   $$  $$<   $$  $$<   $$ |  $$ |
  $$ |$$$$ |$$$$$/    $$$$$  \  $$$$$  \  $$ |  $$ |
  $$ \__$$ |$$ |_____ $$ |$$  \ $$ |$$  \ $$ \__$$ |
  $$    $$/ $$       |$$ | $$  |$$ | $$  |$$    $$/
   $$$$$$/  $$$$$$$$/ $$/   $$/ $$/   $$/  $$$$$$/

	Gekko v0.4.6
	I'm gonna make you rich, Bud Fox.

2017-04-02 19:00:54 (INFO):	Setting up Gekko in realtime mode
2017-04-02 19:00:54 (INFO):
2017-04-02 19:00:54 (INFO):	Setting up:
2017-04-02 19:00:54 (INFO):		 Candle writer
2017-04-02 19:00:54 (INFO):		 Store candles in a database
module.js:471
    throw err;
    ^

Error: Cannot find module '/home/nargren/Software/gekko/core/../plugins/undefined/writer'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at load (/home/nargren/Software/gekko/core/pluginUtil.js:90:25)
    at /home/nargren/Software/gekko/node_modules/async/dist/async.js:1156:9
    at replenish (/home/nargren/Software/gekko/node_modules/async/dist/async.js:1030:17)
    at /home/nargren/Software/gekko/node_modules/async/dist/async.js:1034:9
    at _asyncMap (/home/nargren/Software/gekko/node_modules/async/dist/async.js:1154:5)
    at /home/nargren/Software/gekko/node_modules/async/dist/async.js:1240:16

I normally give the full path to the config

You can't do that anymore (judging by the new code), I will fix that as some point though.

Error: Cannot find module '/home/nargren/Software/gekko/core/../plugins/undefined/writer'

The format of the config has changed (the adapter section specifically). Please update your config! And you should be good.

Thank you, it worked just fine after updating my config file.


[Feature Request] able to resend code to email

i requesting for able to resend email code. since when i recieved code it said was invalid and wasn"t able resend new code. but browser able login fine.

I've also experienced this issue with the game client attempting to login and being sent an email verification code when having 2FA enabled, VRCX and the game expects you to enter in a 2FA code not the email verification code. I may add an option to enter in a email code instead when 2FA is enabled.


How to cleanup k3s agent manually?

So where is k3s-agent-uninstall.sh?
https://rancher.com/docs/k3s/latest/en/installation/uninstall/
I run k3s agent on my host, then all of my web service down.  It seems that it changed my ipatbles a lot, so I'm looking for how to cleanup it.

I try to make workaround for install k3s scripts, it works.

Do you know which github repo I need to fork to do a PR on k3s.sh script ?


Force computation completion

Is there a way to programmatically force Legate to wait for the completion of all pending operations? In the examples, the way to go is to read the output basically, i.e., assert not math.isnan(np.sum(output)). Is there a different way that doesn't incur the penalty of accessing all output elements?

We'll work to add a more ergonomic way to do this in the future. In the meantime, you can use the timing submodule` from the legate core to get the "current time":
https://github.com/nv-legate/legate.core/blob/master/legate/timing/timing.py#L143
If you then print out the time value you receive back (or just assert that is not zero or anything that will force it to be evaluated), that will actually wait for all computations outstanding to finish before performing the timing measurement (you can see the execution fence here that enforces that here: https://github.com/nv-legate/legate.core/blob/master/legate/timing/timing.py#L146). It should be a very low overhead way of forcing all prior computations to complete. It's worth noting that Legate is not doing lazy evaluation, we're actually kicking off computations as you ask for them. We're just doing it asynchronously.

This works great, thanks a lot!


Fix ansible complaining of deprecation warning

Hello,
This PR fixes the following warning:
TASK [openmicroscopy.selinux-utils : system packages | install selinux utilities] **************************************************************************************************************************************************************************************************************************************************************************************************
[DEPRECATION WARNING]: Invoking "yum" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['libselinux-python', 'libsemanage-python', 'policycoreutils-python']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.```

@vulturm in addition to the changes requested by https://github.com/openmicroscopy/ansible-role-selinux-utils/pull/5#pullrequestreview-205107669, it looks like the Travis CI build did not get triggered on this Pull Request.
We are still investigating the reason. Before pushing your commits, could you try logging once in to https://travis-ci.org/ with your GitHub user?

Hi @sbesson ,
Indeed. In my travis account it says that it was flagged for abuse. I've sent an email to support@travis to find out more about it. If there's a chance you could manually trigger it by yourself, please do.
Thanks!

Hi @vulturm we tried to retrigger the Travis build manually a few times but got the same message as the one you received. I also emailed support@travis which led to my comment on https://github.com/openmicroscopy/ansible-role-selinux-utils/pull/5#issuecomment-465301697.

Hi @sbesson ,
I've received an response from the travis-ci team, and they've un-flagged my account.
I've amended the last commit to change it's hash and force pushed it to trigger the build.

@sbesson Tag as 2.0.0?

Because of the Ansible version bump? Most changes look fully backwards-compatible otherwise.

Independently of the discussion about versoning, merging. Thanks @vulturm, we'll let you know when this is deployed to Galaxy

Thank you, too!


Update troubleshoot-exploit-protection-mitigations.md

EAFModules should be deleted too

Docs Build status updates of commit c8ef486:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/defender-endpoint/troubleshoot-exploit-protection-mitigations.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 5c94090:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/defender-endpoint/troubleshoot-exploit-protection-mitigations.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 5c94090:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/defender-endpoint/troubleshoot-exploit-protection-mitigations.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Remove install yarn from GitHub Action example

This removes the step to install yarn, as it appears that's already installed with actions/setup-node@v1. See the section about ]publishing with yarn in the README](https://github.com/actions/setup-node#usage).

Oh cool - I hadn't realised that. That's handy.


Question - Recommendation to Disable SSPR for admins

💡 Summary
We received a recommendation via a comment from an internal CISA reviewer to disable the self service password recovery (SSPR) for administrators. The comment stated that organization cannot manage which specific SSPR authentication methods are available (i.e. you cannot turn off specific ones).
Rationale
The reviewer was concerned about the fact that you cannot control which SSPR methods are available.
Summary of Prototyping
We conducted some hands-on prototyping during #244 and part of that examined the SSPR features specific to administrators.

For admins, the SSPR feature is enabled by default even if the SSPR tenant radio button is Off.
I confirmed that there does not seem to be a way to manage which specific SSPR methods are available for admins.
I also found that in some tenants, admins must use 2 recovery methods to reset their password but in other tenants only 1 method is required to reset.

Our Recommendation
I do not recommend a new policy to disable SSPR for admins.

I think that is too drastic and might lead to unintended consequences. There may be situations like emergency access that could be negatively impacted by completely turning off the ability to perform a self-service password reset.
Also, separate from this comment, we are prototyping the new authentication strength feature to enforce phishing-resistant MFA for admins.
We also have a recommendation to create a new baseline policy "Non phishing-resistant authentication methods SHALL be disabled" (GitHub issue #251) which would restrict which auth methods are available in the tenant.

Questions to CISA

Does CISA recommend to disable SSPR for Admins?
If yes, what is the rationale?

I concur with your recommendation Ted.

Based on the direction from CISA, I'm closing this issue with no further action at this time.