RLN rate-limiting over multiple shards

Background
Waku currently uses RLN rate-limiting to prevent publishers overwhelming the network with high-rate publishing. The current approach is summarised in the Waku Network specification. Of particular interest is how RLN proofs are validated: instead of slashing rate-limit violators' memberships, the validating RLN Relay nodes simply penalise peers forwarding messages exceeding the rate limit by lowering their gossipsub score (which will eventually lead to disconnection). This incentivises all RLN Relay nodes to validate and drop violating messages or else risk their gossipsub score falling to the point where they lose all connectivty.
Problem
An RLN membership allows a publisher to publish messages to multiple shards. Validating nodes record nullifiers included in the RLN proofs over an epoch and detect rate violators by detecting a repeat of a nullifier (see this description). However, if a validating node is subscribed only to a subset of shards it may not have recorded a previous use of a nullifier on an unsubscribed shard, thereby incorrectly forwarding a violating message as valid. This could lead to its gossipsub score being lowered and its disconnection from the network.
Possible solutions
Option 1: Membership per shard (discarded)
Discarded for not being scalable or economic.
Option 2: Include shard in RLN proof and rate-limit per shard

Include the shard in the RLN signal similar to what was proposed for timestamp in https://github.com/waku-org/nwaku/issues/2972
Update validation rules to cache nullifiers per shard and only consider a message invalid if double-signalling detected in a nullifier on the same shard. In other words, the rate limit remains fixed for a single membership valid on all shards, but it becomes a per-shard rate-limit.

Immediate comments on the above:

On second thought, I'm not exactly sure why (1) is necessary? In the case of the timestamp it makes sense to include it in the proof as the timestamp is a spoofable field in the WakuMessage. Publishers cannot "fake" the shard on which they publish a message, so if the validators want to validate per-shard they can already do so by simply considering the shard on which each message arrive?
Afaiu adding the shard to the RLN signal will not alter the resultant nullifier. In other words, for messages generated by the same membership, in the same epoch and that violate the rate-limit (regardless of destination shard) a double nullifier will exist (on some shard). This means the RLN verification rule as described here will be violated. We can of course decide that these messages are not invalid as they "only" double a nullifier on another shard, a condition we now consider valid. However, is it wise to remove RLN's verification as primary rate-limiting check by adding a conditional validation rule on top? Presumably the publisher will simultaneously generate enough key shares for its secret key to be reconstructed (which would lead to slashing, if we had it implemented).

Option 3: Implement slashing
Remove gossipsub scoring as primary disincentivisation mechanism and implement slashing. If even one validating node is subscribed to all shards used by a rate-violating publisher, the publisher will be slashed and its membership removed.
cc @alrevuelta

if a validating node is subscribed only to a subset of shards it may not have recorded a previous use of a nullifier on an unsubscribed shard, thereby incorrectly forwarding a violating message as valid.

That's bad but can be solved by adding the shard to the hash computation of each messages and only allowing shard-specific knowledge for message validation.
This would mean that RLN is no longer a GLOBAL rate-limit is that ok?
Also, option 3 only solve the problem IF it is economically viable to run a node to check ALL messages and slash offenders. It also becomes less and less viable the more shards and traffic increases.

Comment from @alrevuelta on Discord:

Not sure "to cache nullifiers per shard" is needed. I mentioned the timestamp as an example, but we already have two examples, the content topic and the message hash.
https://github.com/waku-org/nwaku/blob/c5a825e206c1cb3e6e0cf8c01410527804cb76c4/waku/waku_rln_relay/rln_relay.nim#L291-L292

My idea is to bind the message to a given shard. This would also fix the meta problem of replaying attacks across shards, which I think we have. A receiver will check the proof constructing a signal with its local shards (due to consistent hashing, it can be taken from the content topic). If the message was meant to another shard, it wont verify.

I think this fixes both replaying attacks and gives determinism to message validation, since it will no longer depend on the shards a node is subscribed to. As a side effect, it makes the rate limit a shard-rate-limit rather than a global-rate-limit.

Re slashing. Agree it can fix the issue but I never liked the fact that going +1 message over the rate limit can kick you out, especially if we have other punishment options. but just an opinion.

Thanks for your comments, @alrevuelta and @SionoiS. I realise there might be something fundamental that I'm missing. 😅
I can see how the message and proof can be tied to a shard by including it in the RLN signal. However, I don't see how this will affect the fact that a publisher is guaranteed to generate a nullifier that has been generated before if it exceeds the shard-agnostic rate-limit.
In other words, you are guaranteed to generate a double nullifier if you generate a proof while

using the same sk
over the same epoch
more than rate-limit times

This is true, no matter how you modify the RLN signal. My understanding of the RLN signal is simply to further prove that this nullifier was indeed computed for the unique message and field values hashed into the signal (so that proofs cannot be reused but are tied to specific messages).
In other words, a validating node may receive a message that doubles a nullifier on some shard, which in RLN rules should count as a violation. I can see how we can choose to ignore it by determining that these double nullifiers were computed for two messages on two different shards, but this would be the wrong way to go IMO.

In my mind the first step for RLN message validation is to check that the hash indeed match the message then you check the zk proof.
in the context of local-only verifiers, they cannot know that a message was sent on other shards BUT they can know that a message was sent multiple time on their shard if we include this information as part of the message hash. This prevents verifiers from spreading invalid messages on their shard.
As for the global rate limit it is impossible for a local verifier to know it was violated. If we want to punish that, slashing should be implemented which may incentivize global verifiers.
As a side note, this here problem is why one should be careful when using a logically centralized system as part of their design, it clash with the other decentralized parts.

As for the global rate limit it is impossible for a local verifier to know it was violated. If we want to punish that, slashing should be implemented which may incentivize global verifiers.

Also agree with the first part. That's why I'm suggesting to apply per-shard rate limits so that "everyone can be in the same page".
Agree also that with slashing this will be fixed, but unsure if the other things slashing introduce are okay.

short term deduplication in gossipsub (~2 mins)

But this protects within the shard. A cross-shard replay attack wont be protected by the gossipsubs cache?

Right! Thanks for all the clarifications. I think we have identified two issues here:
(1) is related to duplicating messages across shards and
(2) about validators with "partial shards view" when the same publisher (sk) uses more than one shard in an epoch
Afaics the only solution for (2) would be slashing. We can continue tracking this problem in this issue.
For (1), encoding the shard into the signal/hash (and validating) would solve the issue. I've created a separate issue to track this: https://github.com/waku-org/research/issues/110


Weird screen bug after installing cesium

After I installed Cesium, I opened the application, then quit it, then the calculator screen itself had some weird bug after

.

You might need to zoom in on the screen its a bit small

I cannot reproduce with latest Cesium. What version are you using?

Okay I can reproduce now :)

Probably fixed with b7eb747


Fix issues with /equities/equity-tickers endpoint and address TPIP security vulnerabilities.

Issue Fixed: #40
This PR addresses and resolves issue #40 related to the [gathering list of nse tickers].
Additional Changes:

Security Vulnerability Fixes: Upgraded key dependencies to mitigate known vulnerabilities:

Upgraded fastapi to 0.114.2
Upgraded requests to 2.32.3
Upgraded numpy to 1.26.4

Ignoring build failure since it seems to be an unrelated issue that needs to be fixed in a new PR.
Tested functionality of all endpoints, none affected.


Add documentation for TensorBoard logging

I could use some help filling in the one-liner for Train/Samples/elapsed_time_ms_backward_inner and Train/Samples/elapsed_time_ms_step.

Thank you @manuelciosici for this contribution, this is very helpful! We were just discussing that this feature was currently undocumented.

elapsed_time_ms_step is the optimizer step time
elapsed_time_ms_backward_inner is the backward time that does not include the the gradient reduction time. Only in cases where the gradient reduction is not overlapped, if it is overlapped then the inner time should be about the same as the entire backward time.

@jeffra Thanks for the help. I copy-pasted your descriptions into the table. I also added the TensorBoard section to the navigation menu.


padRight is undefined

Thanks for the lib!
In docs it says "padRight", but package only exports "padEnd", so one of them needs to be updated, cheers!

Fixed, thank you 👍


Git commit -m closes the terminal using zsh + oh my zsh

Dears,
I have a problem in terminal WSL, when i type git commit -m the terminal is closed!
Note: In the default terminal, this error does not happen. The error happens when I install zsh + plugin oh my zsh
sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
My windows build is Windows 10 Pro, 1903.
I have been installed Ubuntu 18.04 LTS.
Can you help me please?

We have faced the same issues... but with some tests we noticed that the following plugin was causing it: zdharma/fast-syntax-highlighting
Do you also have it installed?

Does this repro in both the Windows Terminal and a conhost (cmd.exe, powershell.exe, wsl.exe) window?
Does this repro in both bash and zsh?

In the WSL happens.
In the others not happens.
@zadjii-msft

having the same issue, commenting to follow

For the record, this is a perfectly good button for following an issue without notifying everyone else on the thread already:

@mufeez-amjad @jonatasbr @eullercdr If you're seeing this, could you follow the Feedback Hub instructions the bot posted? We need more information to be able to investigate this, since we can't repro this on our end.


Set sudo to false in travis config

Setting sudo to false in the travis config will allow the travis tests
to run in travis' container based infrastructure and have faster start
times.

Based on https://docs.travis-ci.com/user/migrating-from-legacy/?utm_source=legacy-notice&utm_medium=banner&utm_campaign=legacy-upgrade

It didn't look like fetch-configlet or configlet needed root to run.

Yeah, they don't need root. Thanks!

PS I've got a branch that actually runs tests. Hopefully I'll get chance to work on it and make a PR soon.

Yay, wouldn't that be awesome!


Add easy Snapdragon upload

This adds a a target ```make eagle_default```` to build both the POSIX and
the QURT side in one command. Also, it adds an upload target for both to
push the files over adb to the device. This doesn't just push the
executables and lib files, but also the startup config files.

@julianoes Tested, works, applied, thanks, closing!

Yay, thank you.


The ressurection of: API Versions (v2)

Changes since v1:

Rebased.

code style will breaks in code that I did not touch

+1 for sol-memmap commit

@cabelitos just a few comments
It's good overall

@bdilly Merged with the mavlink fix.


[8.x] Improve support for custom Doctrine column types

Context
This PR is my and @TomHAnderson's follow-up to several other issues and PRs that address registering custom Doctrine columns in Laravel.
Using custom Doctrine column types in a Laravel app is useful because it makes renaming and changing columns in migrations really easy. While Laravel handles creating columns itself internally, it relies on Doctrine to rename columns or make changes to them. Doctrine doesn't have built-in support for all of Laravel's column types, so in a fresh Laravel app there are actually several column types that can be created fluently in migrations but require custom SQL statements to change or rename later.
The TimestampType added by @TomHAnderson in #35591 is a great example of this—it adds platform-specific SQL statements to support handling timestamp columns on several different database drivers, which allows developers to keep calling ->timestamp() in migrations and always have it behave as expected regardless of which driver they're using. Without it, renaming or changing a timestamp column in a migration might also silently convert it to a datetime column, because that's the closest type Doctrine has built in (see #39959).
The Xml and Bit types used in this PR's tests are similar, platform-specific examples, which would make it possible to do things like $table->renameColumn('old_xml_name', 'new_xml_name') or $table->bit('bit_column')->nullable()->change().
See: #35498, #39959, #35591, #39683.
Motivation
There is existing functionality in the framework that handles most of the required setup for registering custom Doctrine types already, and this PR fills in the gaps to make this functionality more consistent.
Specifically, the Laravel documentation includes an example of registering custom column types automatically by listing them in config/database.php, but this currently doesn't work in all contexts (multiple connections, tests, etc.):
use Illuminate\Database\DBAL\TimestampType;

'dbal' => [
    'types' => [
        'timestamp' => TimestampType::class,
    ],
],

Solution
This PR ensures that all custom Doctrine database column types listed in config('database.dbal.types') are registered consistently with the framework, so that columns using those types can be created, changed, renamed, and deleted in migrations and in tests.
Changes

Moves the automatic registration of custom types in config files from the framework's core DatabaseServiceProvider into the DatabaseManager class at the point where a particular database connection is actually created.
Moves the code responsible for performing the registration of a specific custom type from the Schema Builder class into the database Connection class, because Doctrine's type "mappings" are connection-specific.

Usage
Using the database.dbal.types config array as shown above, or calling DB::connection()->registerDoctrineType() directly, should now work as expected in all contexts, and in tests.
Tests
We added tests for the existing TimestampType verifying that timestamp columns can be modified and/or changed to datetime, and vice versa, on supported database drivers.
We also added tests for completely new custom types, xml in Postgres and bit in MySQL, verifying that:

custom types can be registered automatically simply by adding them to the database.dbal.types config array
custom types can be registered on multiple database connections

Finally, we added a test to verify that custom type registration works in tests.
Note: since this feature relies on data types that are specific to certain database drivers, some of our tests are also slightly different for different databases.

I am not against your pull request but wouldn't it make more sense just providing the missing doctrine types than rewriting parts of Laravel, so the user can add new doctrine types? I mean it's already possible to add custom doctrine types for migrations. I am doing this for PostgreSQL where I am adding numerous missing types [1] [2].

@tpetry yeah providing more of the missing types out of the box would be great! You're right it's already possible to do this, this PR just aims to make it a bit easier.

Using custom DBAL types via database.dbal.types leads to the following error with Laravel's parallel testing feature: Doctrine\DBAL\Driver\PDO\Exception: SQLSTATE[08006] [7] FATAL:  sorry, too many clients already. Tests work as expected when I register the custom DBAL type using DB::registerCustomDoctrineType() in the AppServiceProvider's registermethod.

@skollro Interesting, do you get that error immediately, like on one of the first tests, or only after a lot of tests have run? I wonder if the code we're calling inside Doctrine is creating new connections every time and if there's a way around that.... I'll look into this asap.

@bakerkretzmar Thanks a lot! Yes, it happens after a few tests. I also had the same thoughts and I've tracked it down to this line https://github.com/laravel/framework/blob/8.x/src/Illuminate/Database/Connection.php#L1038

Tests work as expected when I register the custom DBAL type using DB::registerDoctrineType() in the AppServiceProvider's register method.

@skollro can you show me exactly how you're doing this? It fails both ways for me.
I think the line you referenced is causing the tests to create two database connections when they only need one, but I'm not sure why that matters. Laravel is already creating connections and that doesn't cause any issues, so maybe they're getting cleaned up somehow and Doctrine's aren't?

This change broke our tests when we upgraded to 8.78.1
We had dbal array in our config/database.php Removing that and adding a DB::registerDoctrineType() in AppServiceProvider instead fixed it.
 It's working fine with 8.77

Why would I? It's working fine with 8.77

I ask because the error was a MySQL error.  However, I don't want to discount your report to just be the fault of another service.  Can you provide a unit test that replicates the issue you've reported?  I'm afraid that without a test to duplicate it we're going to be at a loss.

@canvural are you running tests in parallel? If so, do you still see those errors when you run them 'normally' (not in parallel)?

@canvural are you running tests in parallel? If so, do you still see those errors when you run them 'normally' (not in parallel)?

No, I don't run parallel.

@bakerkretzmar You're right, I've recreated the issue in a fresh Laravel project, using a PostgreSQL database. Calling DB::registerDoctrineType() in the AppServiceProvider only works for sequential test runners. Not even the test databases are created when parallel testing is used. Maybe @nunomaduro could have a look to help us here?

Also adding DB::registerDoctrineType call to a service provider causes Laravel to make db connections at boot. I'm trying to avoid doing any db calls in any service provider.
An application should be bootable even if there is no db connection or it has wrong configuration.

I created an example repo: https://github.com/canvural/doctrine-type-bug
In the master branch there is just the dbal array in config/database.php And when running tests it fails.
In fixed branch I removed the dbal array and instead added DB::registerDoctrineType to AppServiceProvider and it works.


Add tests for character_activator.0.dart API example.

This PR contributes to https://github.com/flutter/flutter/issues/130459
Description

Adds examples/api/lib/widgets/shortcuts/character_activator.0.dart test
Fixes the description of the examples/api/lib/widgets/shortcuts/character_activator.0.dart example

Pre-launch Checklist

[X] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[X] I read the [Tree Hygiene] wiki page, which explains my responsibilities.
[X] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
[X] I signed the [CLA].
[X] I listed at least one issue that this PR fixes in the description above.
[X] I updated/added relevant documentation (doc comments with ///).
[X] I added new tests to check the change I am making, or this PR is [test-exempt].
[X] I followed the [breaking change policy] and added [Data Driven Fixes] where supported.
[X] All existing and new tests are passing.

@Renzo-Olivares @dkwingsmt thanks for the review!


Here's some examples of promQL:
To simply get http calls count:
netobserv_http_api_calls_duration_count

To get P50, P75 and P90 of parallelized loki calls
histogram_quantile(0.5, sum(rate(netobserv_loki_parallel_calls_duration_bucket{}[1m])) by (le))
histogram_quantile(0.75, sum(rate(netobserv_loki_parallel_calls_duration_bucket{}[1m])) by (le))
histogram_quantile(0.9, sum(rate(netobserv_loki_parallel_calls_duration_bucket{}[1m])) by (le))

To get P90 of HTTP calls per handler and code:
histogram_quantile(0.9, sum(rate(netobserv_http_api_calls_duration_bucket{}[1m])) by (le, handler, code))

Picture shows the stacked P50/P75/P90 :

cc @memodi

/approve


Improved Phased Assembler failing at ovl_filt

Hi there-
I am attempting to use IPA on our RHEL 7.6 HPC and Slurm.
As a test, I have used an RSII dataset I have laying around.
Package name
┌─[09:49:25]─[nblouin@tlog2]─[~/scratch/ModuleTests/IPA]
└──> ipa --version
ipa (wrapper) version=1.0.3
'''
Describe the bug
All appears to be running OK up to the point of ovl_filt where I get a warning:
"Bad value: "--max-diff" for option "max-cov"; expecting int" and then:
"Error in rule ovl_filter".
From looking at the call (portion of the out file pasted below)  it appears the pipeline is setting a value for --max-diff and --max-cov twice in the same call and that I do not have control over that step. Any advice would be great.
Error message
...
....
Computed maximum allowed coverage:

ln -sf ../phasing_merge/ovl.phased.m4 .
++ basename ../phasing_merge/ovl.phased.m4

local local_m4=ovl.phased.m4

IPA_TIME log.ovl_filter.m4filt.memtime falconc m4filt --n-proc 16 --in ovl.phased.m4 --out ovl.chimerfilt.m4 --keepIntermediates --max-diff 80 --max-cov 100 --min-cov 2 --bestn 10 --min-len 4000 --gapFilt --minDepth 4 --idt-stage2 98 --max-cov --max-diff --filter-log m4-readfilt.log

local local_out_memtime=log.ovl_filter.m4filt.memtime

shift

[[ Linux == Linux ]]
+ /usr/bin/time '--format=cmd: %C\nreal_time: %e s\nuser_time: %U s\nsys_time: %S s\nmax_rss: %M kB\nexit_status: %x' -o log.ovl_filter.m4filt.memtime falconc m4filt --n-proc 16 --in ovl.phased.m4 --out ovl.chimerfilt.m4 --keepIntermediates --max-diff 80 --max-cov 100 --min-cov 2 --bestn 10 --min-len 4000 --gapFilt --minDepth 4 --idt-stage2 98 --max-cov --max-diff --filter-log m4-readfilt.log
Bad value: "--max-diff" for option "max-cov"; expecting int
m4filt [required&optional-params]
Run the multi-stage m4 overlap filter (for HiFi Asm). Take only one m4 file,
inFn. If in+'.idx' exists, that is the index. In stage one, reads that trigger a
filter are marked including containment, gaps in coverage along the A-read, and
repeat reads. In stage two the filters are applied and the N-best overlaps are
kept for the 5prime and 3prime of each read.
Options:
-h, --help                                 print this cligen-erated help
--help-syntax                              advanced: prepend,plurals,..
-i=, --idt-stage1=       float   90.0      Stage one percent identity filter,
formatted as percentage, overlaps <
%idt are skipped
--idt-stage2=            float   90.0      Stage two percent identify filter
-m=, --min-len=          int     6000      Minimum read length, reads shorter
than minLen will be discarded
--min-cov=               int     2         Minimum number of overlaps on
either side of a read
--max-cov=               int     200       Maximum number of overlaps on
either side of a read
--max-diff=              int     100       Reads are skipped is abs(5p-3p)
overlap counts > maxDiff
-b=, --bestn=            int     10        Keep N best overlaps at 5prime AND
3prime of a read
--min-overhang=          int     0         Minimum amount of overhang bases in
an overlap to keep the overlap
--min-depth=             int     2         Depths lower than minDepth are
considered gaps
-g, --gap-filt           bool    false     Run depth filter, takes a little
more time
-k, --keepIntermediates  bool    false     set keepIntermediates
-n=, --n-proc=           int     24        Number of processes to run locally
--in-fn=                 string  REQUIRED  M4 overlaps file
-f=, --filter-log-fn=    string  REQUIRED  Write read filter stats to this
file
-o=, --out-fn=           string  REQUIRED  Final m4 overlaps file
0.04user 0.04system 0:00.09elapsed 91%CPU (0avgtext+0avgdata 6940maxresident)k
0inputs+0outputs (0major+8778minor)pagefaults 0swaps
[Fri Jun 26 10:20:36 2020]
Error in rule ovl_filter:
jobid: 7
output: ovl_filter/ovl.final.m4, ovl_filter/ovl.chimerfilt.m4
shell:
  wd=$(dirname ovl_filter/ovl.final.m4)
  mkdir -p $wd
  cd $wd
  rel=..

  input_m4="$rel/phasing_merge/ovl.phased.m4"         output_m4_final=$(basename ovl_filter/ovl.final.m4)         output_m4_chimerfilt=$(basename ovl_filter/ovl.chimerfilt.m4)         params_num_threads=16         params_config_sh_fn="$rel/generate_config/generated.config"         params_log_level="INFO"         params_tmp_dir="/tmp"             time ipa2-task ovl_filter

  (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /pfs/tsfs1/gscratch/nblouin/ModuleTests/IPA/GORYtester/.snakemake/log/2020-06-26T095853.691230.snakemake.log
CWD:/pfs/tsfs1/gscratch/nblouin/ModuleTests/IPA/GORYtester
NPROC:16
NPROC_SERIAL:16
config:{'advanced_options': '', 'coverage': 0, 'genome_size': 0, 'max_nchunks': 40, 'nproc': 16, 'phase_run': 1, 'polish_run': 1, 'reads_fn': 'GORYtester/input.fofn', 'tmp_dir': '/tmp'}
Done
To Reproduce
Here is how I am running IPA:
**#!/bin/bash
#SBATCH --job-name t-IPA
#SBATCH --nodes=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=0
#SBATCH --time=108:00:00
#SBATCH -D /gscratch/nblouin/ModuleTests/IPA
echo "SLURM_JOB_ID:" $SLURM_JOB_ID
echo "SLURM_JOB_NAME:" $SLURM_JOB_NAME
echo "SLURM_JOB_PARTITION" $SLURM_JOB_PARTITION
echo "SLURM_JOB_NUM_NODES:" $SLURM_JOB_NUM_NODES
echo "SLURM_JOB_NODELIST:" $SLURM_JOB_NODELIST
echo "SLURM_JOB_CPUS_PER_NODE:" $SLURM_JOB_CPUS_PER_NODE
echo "SLURM_TASKS_PER_NODE:" $SLURM_TASKS_PER_NODE
echo "SLURM_CPUS_PER_TASK:" $SLURM_CPUS_PER_TASK
echo "SLURM_CPUS_ON_NODE:" $SLURM_CPUS_ON_NODE
echo "Starting"
module load  singularity/2.5.2 ipa
echo "Modules loaded:"
echo "Starting IPA"
srun ipa local --verbose --run-dir GORYtester --nthreads 16 --njobs 1
-i XNBUW_20181115_S54024R1_PL100107678-1_C01.subreads.fastq.gz
-i XNBUW_20181115_S54024R1_PL100107678-1_B01.subreads.fastq.gz
echo "Done**
Expected behavior
A clear and concise description of what you expected to happen.

I'm guessing this call failed:
falconc ovl-cov-stats --in-fn=ovl.fofn
can you check the content of
covstat.json in the same folder.

Thanks for the reply.
Let me know if you need to peek at any other files.
Here you go:
┌─[14:22:03]─[nblouin@tlog2]─[~/scratch/ModuleTests/IPA]
└──> cat !$
cat GORYtester/ovl_filter/covstat.json
{
"mean": 0.0,
"std": nan,
"median": 0.0,
"min": 0,
"max": 0,
"n": 0
}

Hi @nblouin,
I assume your RSII dataset is CLR and not >Q20 CCS (i.e. HiFi)?
IPA is intended for HiFi reads of very high accuracy, so CLR would not work with it.
Best regards,
Ivan.

Hi Ivan-
My mistake. I thought this test set was HiFi (I only looked at the run time and assumed it was not a CLR library as it ran for 30 hrs).
I have now tested my installation  with a known HiFi set of reads and it ran without a hitch.
Thanks for the help and apologies for the rookie error.
┌─[10:55:57]─[✗:5]─[nblouin@tlog2]─[~/scratch/ModuleTests/IPA/SRR11606869Tester]
└──> tail /pfs/tsfs1/gscratch/nblouin/ModuleTests/IPA/SRR11606869Tester/.snakemake/log/2020-06-30T091952.143514.snakemake.log
[Tue Jun 30 18:53:08 2020]
localrule finish:
input: final/final.p_ctg.fasta, final/final.a_ctg.fasta
jobid: 0
reason: Input files updated by another job: final/final.p_ctg.fasta, final/final.a_ctg.fasta
[Tue Jun 30 18:53:08 2020]
Finished job 0.
52 of 52 steps (100%) done
Complete log: /pfs/tsfs1/gscratch/nblouin/ModuleTests/IPA/SRR11606869Tester/.snakemake/log/2020-06-30T091952.143514.snakemake.log

Thanks for posting. We should fail early on CLR input.


Blockchain Public Address API - Aligment with Commonalities and ICM MetaRelease Spring 25 Guidelines

Problem description
Aligment with Commonalities and ICM is needed. To do so Check the Commonalities and ICM scope issues and related release PRs to ensure alignment with their upcoming versions
Parent Issue: #61
Possible evolution
Based on upcoming merged PRs identify the related impacts in Blockchain Public Address API and align the design accordingly
Alternative solution
N/A
Additional context
It is needed to have some PRs merged in order to begin this work

NOTE: Alpha Release for Commonalities will be expected soon, so as we have the first set of PRs to evaluate


Enhance docs with possible values of Compute operation's .error.errors[].code

We should figure out possible values that can be assumed by error.errors[].code.
The field is only documented as:

The error type identifier for this error.

This has been added to our feature backlog: https://github.com/GoogleCloudPlatform/google-cloud-java/wiki/Feature-backlog . This issue will be closed but is linked in the backlog and can continue to be used for comment and discussion.


Ft/documentation

WHAT DOES THIS PR DO
Documenting pcb-design, product specification and troubleshooting using mkdocs

One more comment, I added EditorConfig linter to the docs as well. If you could re-base your branch to pull in the changes 😉


Get rid of references to --cap-remove, --cap-retain in the documentation

xref https://github.com/coreos/rkt/pull/3192#issuecomment-247318724

Just want to point out that cap-remove and cap-retain flags are still available as not-deprecated flags in prepare - https://github.com/coreos/rkt/blob/cf0aa2e4c6062e76c6a18a8af99641e84aac9ecc/rkt/prepare.go#L76.
Likely @alban will fix this as a part of #3207.

Fixed by https://github.com/coreos/rkt/pull/3209.


Performance issue with large ranges

Each charCode in a range is added to the output structure
This means that given a large range such as
/[\u0240-\uffff]/ // used in CSS 2.1 in Identifiers

An array of ~65k numbers is created.
This is obviously incredibly inefficient.
Sets should be represented more efficiently.

this is resolved in 0.3.5


Loss function when using custom eval_metric

Hi,
I'm trying to understand what is happening behind the scene. What is the loss function used by XGBoost when objective = 'binary:logistic', eval_metric=cost, scale_pos_weight=60?
cost = 10*false positives + 500*false negatives
Is the loss function 'merror'...If yes then how is the tree fitted on derivatives of residuals? or Is it logloss...if yes then how is the parameter 'scale_pos_weight' used?

Since you are using objective='binary:logistic', the loss function used for training is the logistic loss. For the use of scale_pos_weight, see https://github.com/dmlc/xgboost/blob/v0.60/src/objective/regression_obj.cc.

Thanks a lot for clarifying my doubt.


fix: add missing json fields for sample streaming http-to-http

What this PR changes/adds
Fixes issues #244 and #334 for sample transfer/streaming/streaming-01-http-to-http

Missing assigner field in contract request
Missing transferType in transfer request
Did not use JSON files in testcase

Why it does that
Sample can not be executed completely.
Further notes
Linked Issue(s)
Closes #244
Closes #334

@ndr-brt:
Since I can't find any doc about how to run the test-cases, I can't fix it that easily.
The official doc returns 404 for almost every link in there (i.e. https://eclipse-edc.github.io/testing.md found in https://eclipse-edc.github.io/documentation/for-contributors/#52-testing-best-practices).
I flew over the convention slightly, does the new name match?

@ndr-brt: Since I can't find any doc about how to run the test-cases, I can't fix it that easily. The official doc returns 404 for almost every link in there (i.e. https://eclipse-edc.github.io/testing.md found in https://eclipse-edc.github.io/documentation/for-contributors/#52-testing-best-practices).
I flew over the convention slightly, does the new name match?

link is broken, but you can find it under contributors manual/writing tests

link is broken, but you can find it under contributors manual/writing tests

Almost all links are broken, unfortunately ;-)
For me, it still does not work for half of the test cases, since Testcontainers does not recognize the docker-engine.
Using the official Docker engine (no Desktop) on Ubuntu 24.04. User is in correct group and can contact the daemon.
Path /var/run/docker.sock exists and has rw access for the group the user is in.
Tried it with setting the DOCKER_HOST variable as well. No change.

link is broken, but you can find it under contributors manual/writing tests

Almost all links are broken, unfortunately ;-)

contributions are the key ;)

For me, it still does not work for half of the test cases, since Testcontainers does not recognize the docker-engine. Using the official Docker engine (no Desktop) on Ubuntu 24.04. User is in correct group and can contact the daemon. Path /var/run/docker.sock exists and has rw access for the group the user is in. Tried it with setting the DOCKER_HOST variable as well. No change.

I'm also using docker engine on ubuntu without issues, it worked out-of-the-box, there's no need to do changes, as long as docker is executable from the command line by a non root user.
what's the error?

link is broken, but you can find it under contributors manual/writing tests

Almost all links are broken, unfortunately ;-)

contributions are the key ;)

For me, it still does not work for half of the test cases, since Testcontainers does not recognize the docker-engine. Using the official Docker engine (no Desktop) on Ubuntu 24.04. User is in correct group and can contact the daemon. Path /var/run/docker.sock exists and has rw access for the group the user is in. Tried it with setting the DOCKER_HOST variable as well. No change.

I'm also using docker engine on ubuntu without issues, it worked out-of-the-box, there's no need to do changes, as long as docker is executable from the command line by a non root user. what's the error?

Strange.
Basically, the long exception-message starts with:
Advanced01openTelemetryTest > runSampleSteps() FAILED
    java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration
        at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$7(DockerClientProviderStrategy.java:274)

It seems to be one of those funny things where installing OS updates and rebooting fixes it.
Can't explain why, since OS was pretty up-to-date and settings seemed right, but it worked.

@ndr-brt: Took me a long time, but I fixed it. Not sure why ECA does not evaluate anymore, or why it evaluated before, since I am using my anonymized Github email. I filled out the form though.

I think it is becuse your first commit, that has no email reported: https://github.com/eclipse-edc/Samples/pull/335/commits/2f00b9d846db09c744125cbdaa25b99bc9dd31f4
maybe you could squash the three commits into one that has the correct info

@ndr-brt:
You mean the last one? Indeed this one was missing the author completely.

Fixed that by a rebase / amend, should be better now.
Haven't seen the checkstyle errors during testing, fixed those as well.
Warnings were there before, I am not about to fix those as well.
So it should be good if all Github checks pass.
Btw: Is there a way to only run a certain test? It was quite annoying, since though I tried some things from the Gradle docs,
I could only get all tests running by firing:
.\gradlew test -DrunAllTests="true"

@ndr-brt: You mean the last one? Indeed this one was missing the author completely.  Fixed that by a rebase / amend, should be better now.
Haven't seen the checkstyle errors during testing, fixed those as well. Warnings were there before, I am not about to fix those as well. So it should be good if all Github checks pass.
Btw: Is there a way to only run a certain test? It was quite annoying, since though I tried some things from the Gradle docs, I could only get all tests running by firing:
.\gradlew test -DrunAllTests="true"

if you are using Intellij you just need to add the -DrunAllTests="true" in the run configuration.
you could also create a configuration template and make it enabled for all the gradle commands (very handy for EDC projects)

@ndr-brt: You mean the last one? Indeed this one was missing the author completely.  Fixed that by a rebase / amend, should be better now.
Haven't seen the checkstyle errors during testing, fixed those as well. Warnings were there before, I am not about to fix those as well. So it should be good if all Github checks pass.
Btw: Is there a way to only run a certain test? It was quite annoying, since though I tried some things from the Gradle docs, I could only get all tests running by firing:
.\gradlew test -DrunAllTests="true"

if you are using Intellij you just need to add the -DrunAllTests="true" in the run configuration. you could also create a configuration template and make it enabled for all the gradle commands (very handy for EDC projects)

But this would still run all tests, doesn't it?
Since it was pretty basic testing so far, the setup is quite simple (no sophisticated IDE).
May have to change that once I start real development.
Checks look good now, right?

if you are using Intellij you just need to add the -DrunAllTests="true" in the run configuration. you could also create a configuration template and make it enabled for all the gradle commands (very handy for EDC projects)

But this would still run all tests, doesn't it? Since it was pretty basic testing so far, the setup is quite simple (no sophisticated IDE). May have to change that once I start real development.

no, this will say to junit: "run the test you want without considering the tag attached to it.

Ok, might work with IDEA. You don't know a way to run a specific test through CLI?

sure, for this one just execute:
./gradlew :system-tests:test --tests "org.eclipse.edc.samples.transfer.streaming.Streaming01httpToHttpTest.streamData" -DrunAllTests=true

Ok, might work with IDEA. You don't know a way to run a specific test through CLI?

sure, for this one just execute:
./gradlew :system-tests:test --tests "org.eclipse.edc.samples.transfer.streaming.Streaming01httpToHttpTest.streamData" -DrunAllTests=true

Haven't tried it yet, but sounds as if it could work. Thank you.
I had tried it basically without the -DrunAllTests=true, since I thought it would do the opposite of what I wanted to achieve.
I read about the fact that otherwise system-tests would not be executed at all, but I thought it would be sort of a side-effect of running all existing test-cases.
Might be a case where we first enlarge the scope to then filter out.
Not too familiar with Gradle unfortunately.

Haven't tried it yet, but sounds as if it could work. Thank you. I had tried it basically without the -DrunAllTests=true, since I thought it would do the opposite of what I wanted to achieve. I read about the fact that otherwise system-tests would not be executed at all, but I thought it would be sort of a side-effect of running all existing test-cases. Might be a case where we first enlarge the scope to then filter out. Not too familiar with Gradle unfortunately.

the -DrunAllTests=true basically says "run the selected tests despite the junit tag", given the fact that this test has the "EndToEnd" label, that property is a sort of passepartout to not care about the labels and just run the tests.
For this test, this command does the same exact thing as the one before
./gradlew :system-tests:test --tests "org.eclipse.edc.samples.transfer.streaming.Streaming01httpToHttpTest.streamData" -DincludeTags="EndToEndTest"

Thank you for the explanation (and preventing me from dying uneducated ;-)).

anyway, thanks for the contribution, I'm gonna merge this 🚀

Perfect, thanks. You are welcome. Hope it helps somebody.


Allow editing multiple roles with one command

Assigned @AlphaStyle since he's working on it already:


Idea crates for next release :-)

Some idears for the next release
Add some breakpoints for viewports:
The breakpoints overwrite the default setting based on viewport width.
var mixer = mixitup(containerEl, {
	pagination: {
		limit: 9,
		maxPagers: 4
	},
	//example
	breakpoint: true,
	breakpoints: {
		1024: {
			pagination: {
				limit: 12,
				maxPagers: 10
			}
		},
		768: {
			pagination: {
				limit: 8,
				maxPagers: 8
			}
		},
		640: {
			pagination: {
				limit: 6,
				maxPagers: 6
			}
		},
		320: {
			pagination: {
				limit: 2,
				maxPagers: 4
			}
		}
	}
});

There's a demo included in the pagination source called "responsive pagination" which shows how to do this.


EL / Greek VRNs do not validate

I've noticed that this tool marks EL-prefixed VRNs as invalid.
The following VRNs are valid according to VIES, but invalid when checked with this validator:

EL027408226
EL032880379
EL800620092

I think the problem is the use of 'GR' in the formatVatNumber $pattern, where 'GR' is used rather than 'EL'.
Fix: $pattern = '/^(AT|BE|BG|CY|CZ|DE|DK|EE|EL|ES|FI|FR|GB|HR|HU|IE|IT|LT|LU|LV|MT|NL|PL|PT|RO|SE|SI|SK)[A-Z0-9]{6,20}$/';

@Kier #11 was merged into master! Thank you very much for your fast fix.
Maybe in later versions I will refactor the code based on this specification: https://www.avalara.com/vatlive/en/eu-vat-rules/eu-vat-number-registration/eu-vat-number-formats.html


Feature: Expose Tx timestamp.

Consider exposing Timestamp https://github.com/corazawaf/coraza/blob/main/internal/corazawaf/transaction.go#L116 in types.Transaction interface.

Do you mind elaborating on the reason? What is the use case?

Do you mind elaborating on the reason? What is the use case?

I was working on TTL based cache for Tx, so having timestamp from actual transaction could benefit by reducing addition calls to time.Now() for getting same information.

@s3rj1k this is a legit use case, do you have any code you could share for such a thing?

Same answer as before, code is not available sadly.


CLI updates for Infracost Cloud with Atlantis, Jenkins and other deployment systems

The Infracost CLI needs to support the following env variables as overrides so Atlantis (and other deployment systems) can provide them manually. We already have INFRACOST_VCS_REPOSITORY_URL, INFRACOST_VCS_SUB_PATH, INFRACOST_VCS_PULL_REQUEST_URL and INFRACOST_TERRAFORM_WORKSPACE. Once the following is done, all top-level run metadata can be overridden with env vars.
INFRACOST_VCS_PROVIDER # validate the value as only github, gitlab, azure_repos, bitbucket are accepted
INFRACOST_VCS_PULL_REQUEST_AUTHOR
INFRACOST_VCS_PULL_REQUEST_TITLE
INFRACOST_VCS_BASE_BRANCH
INFRACOST_VCS_PULL_REQUEST_ID
INFRACOST_VCS_PIPELINE_RUN_ID

INFRACOST_VCS_BRANCH
INFRACOST_VCS_COMMIT
INFRACOST_VCS_COMMIT_TIMESTAMP
INFRACOST_VCS_COMMIT_AUTHOR_NAME
INFRACOST_VCS_COMMIT_AUTHOR_EMAIL
INFRACOST_VCS_COMMIT_MESSAGE

Rename the Infracost JSON metadata vcsRepoUrl to vcsRepositoryUrl. That's the only one that's not consistently named with its env equivalent, INFRACOST_VCS_REPOSITORY_URL.

The Infracost CLI needs to be updated to set Atlantis and CircleCI vcs provider to the actual vcs provider, not "atlantis" or “circleci”. This'll enable Infracost Cloud to render icons for them correctly.

We should update the 3 examples in Atlantis to set the required env vars for Infracost Cloud's new dashboard. Ideally we should do this for GitHub, GitLab, Azure Repos and Bitbucket e.g. for GitHub we need something like this:
INFRACOST_VCS_PROVIDER=github # For GitHub Enterprise, also use "github"
INFRACOST_VCS_PULL_REQUEST_URL=https://github.com/$BASE_REPO_OWNER/$BASE_REPO_NAME/pulls/$PULL_NUM
INFRACOST_VCS_PULL_REQUEST_AUTHOR=$PULL_AUTHOR

INFRACOST_VCS_PULL_REQUEST_TITLE=$(curl -s
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: $GITHUB_TOKEN" \
  "https://api.github.com/repos/$BASE_REPO_OWNER/$BASE_REPO_NAME/pulls/$PULL_NUM" | jq -r '.title')

We should update the Jenkins example to show how the above envs can be set for Infracost Cloud's new dashboard.

Once the above is done, we need to add an explicit way for users to mark the run as complete, or upload the results to Infracost Cloud, then the Infracost Cloud dashboard should work fine.

@hugorut I updated this issue to add VCS_ to branch and commit env vars. Please update the issue desc if I got any wrong as I'll use this issue to write the docs for these envs.

@alikhajeh1 I've actually decided to keep the old non VCS_ branch vars as I think the separation you had before was useful. Let me know if you think there's a strong reason to change to the VCS_ naming for those.

Additional env var support is now merged in master and will be available with the next release. I'm keeping this issue open as it still depends on https://github.com/infracost/infracost/pull/1997 (infracost upload support)

This functionality has been published in the 0.10.11 release.


Remove the archived_pull_requests table

In previous years we didn't have proper scoping of pull requests so we threw old prs from previous years into a table called archived_pull_requests, but now everything is scoped correctly we don't need that for 2016/2015 but all the old prs for 2014/2013/2012 are in the archived_pull_requests table.
We need:

[x] a script to move all the archived_pull_requests back into the pull_requests table https://github.com/24pullrequests/24pullrequests/issues/1727
[ ] a migration to drop the archived_pull_requests table https://github.com/24pullrequests/24pullrequests/issues/1727
[ ] all uses of ArchivedPullRequests to be removed across the codebase and switched back to regular methods on PullRequest https://github.com/24pullrequests/24pullrequests/pull/1729

Running the rake task in #1727 now


[flutter_tools] hide Skia specific rendering options.

At some point in the near future, we'll start an Impeller on Android preivew - and later make it the default. Unlike Skia, Impeller does not have a fallback software rendering mode. We'd like to stop suggesting this as an option now, and in the future remove the option to force software rendering.
Once impeller is the default, asking for software rendering on Android will result in either an error or falling back to Skia.

@christopherfujino not sure what the current strategy for testing these args changes should be. But I could apply for a test exemption if you think that is the best option.

Yeah, you could test that we DON'T print the status about using hardware rendering, however, that seems like a somewhat contrived test so I'd request a NTE.

test-exempt: changes help text in a harmless way and removes code
not sure why the bot didn't flag it though.


Create Windows .exe installer

[x] Windows: Create installer .exe file

[x] Loading .gif while install happens
[x] Install desktop/start menu shortcuts
[x] Installer icon
[x] Prevent --squirrel arguments from getting added as torrents
[x] Handle --squirrel arguments
[x] Fix install gif to not blink

[x] app.setAppUserModelId
[x] Windows: Server-side components for updater
[x] Build final artifacts to dist/ folder instead of inside each platform folder
[x] Simplify arguments to npm run package. Now It's npm run package -- win32
[x] Don't autohide top menu – it's important

Moved the .exe signing out of the tasks for this PR. That can happen later.

This is all working now! (Minus signing the .exe, which I'll do once I have a cert)
Going to do a 0.1.0 release and see how it goes :+1:

Hi !
Impressive work !
Just a very minor issue : during installation, there is no icon in the windows taskbar in the v0.3.3

@trash-anger yes, open a new issue


 This wouldn't allow us to have "futures" as it were, but it would be nice so that we could at least take advantage of the cross platform commands builtin into CMake.
Whether this command is called task(), shell(), or even just cmake() is still TBD and needs some investigation into what feels most ergonomic.
If we could also somehow support pipelines, that would be killer :)

UPDATE: I am currently quite partial to the name system(), as this is intended to be "system agnostic".
I'll be looking into this shortly.

After looking into this, the most I could do is rename execute_process to process, but the effort isn't worth it.


Is it possible to use Modnet with gpu acceleration ?

Hi @ml-evs , @ppdebreuck , I have been trying to use Modnet with GPU. But cannot get it to work. I had to use a different tensorflow version than pinned on modnet. I am using tensorflow==2.15.0. (As using 2.11.0, gpus are not detected at all on my system)
System has cuda 12.4 installed.
It always fails with the following error
"CUDA-capable device(s) is/are busy or unavailable" or failed to set cuda device.
I tested with this to see if the TensorFlow is installed correctly. It seems to be the case tensorflow is working fine.
import os
import tensorflow as tf
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
tf.debugging.set_log_device_placement(True)
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)

with tf.Session() as sess:
    print (sess.run(c))

Not able to figure out what could be the problem here. Any help in this regard would be great!

Could you post a minimum failing MODNet script too? I've never had issues trying it out on my GPU with TF 2.11 (I doubt much of MODNet works with a more recent TF, though we don't really have the resources to spend to update it).

Hi @ml-evs , will post in today in sometime. Need to wait for gpus to be free again.
Can you also share your environment file, I can test with that too if that helps.

I haven't tried for years as I never saw any worthwhile speed up for the size of networks I was creating, so don't have one to hand unfortunately. But I can try to reproduce your issue locally at least.

Hi @ml-evs , I realized the example script is almost same as in one of the example notebooks, just with additional features appended to dataframe .
Just at the top I have this additional lines to limit to 1 GPU
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

https://github.com/ppdebreuck/modnet/blob/master/example_notebooks/training_multi_data.ipynb
On further checking a bit, I find when I try to use GA especially this error occurs. In the same notebook, it seems to run fine untill following line.
# fitting
model.fit(train_data)

But When I try to run with Gentic algo based hp opt, i get cudaSetDevice() on GPU:0 failed. Status: CUDA-capable device(s) is/are busy or unavailable error. So seems somehow gpu gets registered once model is initialized and not get released for next iterations.
I also tried with model.fit_preset(train_data) same error.
Also, just want to mention when I ran this tests , I have ran this fit commands independently after restarting kernel. So gpu was free in all the cases. It worked only when not using preset or genetic algo

Also, just want to mention when I ran this tests , I have ran this fit commands independently after restarting kernel. So gpu was free in all the cases. It worked only when not using preset or genetic algo

I don't have time to look into this fully now, but my guess is that the first preset is running and the rest see that the device is busy, as we are using Python multiprocessing and typically TF will allocate the entire GPU ot the first process. You could try e.g. tf.debugging.set_log_device_placement(True). If this is the case, you might be able to fiddle around with set_memory_growth to allow each process to use a small amount of memory initially. I'm not an expert on this, but in the past the gains we saw were very small (/non-existent) from using a GPU for small MODNet models (which is to say I'd be interested in your results if you can get this working!)

Hey all ! Sorry for the late reply. I would not spent time on this @naik-aakash: (i) MODNet uses small networks (intended for small datasets), with no benefit of GPU training (except if you force on having big architectures, but this would be a very specific use case). (ii) We are not fully happy with TF, and would like to migrate to Torch or JAX. Matthew and I don't have time for this, but we might have a student doing this next semester (finger crossed) ;p

Just to add I'd still love to find time to update MODNet to the latest Keras core, which has torch/Jax and TF backends (and hopefully should be quite an easy translation job!) -- see #158 for some details

Great idea, and probably easier than translating to PyTorch given our frequent usage of tf.keras


How to use css modules

Is there a way to use css modules with docz?
Description
I'm using css modules with the webpack loader css-loader
here is a component for an exemple
import * as React from 'react'
import classNames from 'classnames'

const styles = require('./styles.css')

interface State {
  isFocused: boolean
}

export default class TextField extends React.Component<{}, State> {
  public constructor(props: {}) {
    super(props)

    this.state = {
      isFocused: false
    }
  }

  private onFocus = () => {
    this.setState({ isFocused: true })
  }

  private onBlur = () => {
    this.setState({ isFocused: false })
  }

  public render() {
    return

      <input className={styles.input}
        onFocus={this.onFocus}
        onBlur={this.onBlur} />
      label

  }
}

and my very simple mdx file
---
name: TextField
---

import {Playground } from 'docz'
import TextField from './'

# TextField

<Playground>
  <TextField />
</Playground>

when I start docz dev server, I have this error message
 ERROR  Failed to compile with 1 errors                                                                       09:39:00

 error  in ./src/libs/text-field/text.mdx

Module build failed (from ./node_modules/@mdx-js/loader/index.js):
Error: ENOENT: no such file or directory, open '/Users/bgranger/htdocs/perso/pocs/docz/src/libs/text-field/text.mdx'

 @ ./.docz/app/imports.js 9:11-123
 @ ./.docz/app/index.jsx
 @ multi ./node_modules/babel-polyfill/lib/index.js ./.docz/app/index.jsx

I assume that the css loader is not used with the bundler.
I have two questions:

is it possible to use css modules?
if not is it possible to use custom bundler/bundler config?

You can set your own bundler config by passing on doczrc.js:
// doczrc.js
export default {
  modifyBundlerConfig: (config) => {
    /* logic here */
    return config
  }
}

More info on issue #6

@pedronauck I setup a sass-loader in the doczrc.js but when it came to loading a css-module (albeit in scss), I recieved the error that 'local module MySCSSModule.scsscould not be resolved'. The import wasimport * as styles from "./MySCSSModule.scss"`. This seems like the loader isn't properly configured.
I added my config by pushing to the config.modul.rules array; I'm assuming that this is the WRONG way and that I didn't configure properly, but the docs are particularly sparse on this detail. Is it possible to have an example of adding config here please?

Try to config like that:
// doczrc.js
export default {
  modifyBundlerConfig: (config) => {
    config.resolve.extensions.push('.scss')
    config.module.rules.push({
      test: /\.scss$/,
      use: ["style-loader", "css-loader", "sass-loader"]
    })

    return config
  }
}

After released v0.2.11 you can use docz-plugin-css to configure your CSS just by adding a plugin. ✌

Nice work. Thanks.


LineChart displays incorrect graph

The FlSpots are:
[
    FlSpot(1.0, 2.0),
    FlSpot(16.0, 1.0),
    FlSpot(17.0, 1.0),
    FlSpot(18.0, 2.0),
    FlSpot(19.0, 3.0),
]

And the widget appears:

But I expected this:

This is the full code:
LineChartData(
        gridData: FlGridData(show: false),
        titlesData: FlTitlesData(show: false),
        borderData: FlBorderData(show: false),
        maxY: 4,
        minY: 0,
        maxX: 31,
        lineBarsData: <LineChartBarData>[
          LineChartBarData(
            spots: [
    			FlSpot(1.0, 2.0),
    			FlSpot(16.0, 1.0),
   				FlSpot(17.0, 1.0),
    			FlSpot(18.0, 2.0),
    			FlSpot(19.0, 3.0),
            ],
            isCurved: true,
            colors: <Color>[DefaultColorTheme.main],
            barWidth: 4,
            isStrokeCapRound: true,
            dotData: const FlDotData(
              show: false,
            ),
            belowBarData: BarAreaData(show: false),
          ),
        ],
)

Hi @imaNNeoFighT, This occurs not only these spots, but also other many spots.
Is this related to the isCurved option?

Thanks!


add maybe package

This is an initial proposal for a helper package as requested so that #26 can be unblocked. The naming of the package is of course contentious, I'm happy to change it to whatever the maintainer prefers.

Feedback addressed and commits squashed. Happy to iterate further on this if needed :)


Dropdown: optionally add dropdown-toggle class

Should fix #162

Can you please fix tslint err? So I can merge it? :)

[Current coverage][cc-pull] is 81.35% (diff: 100%)

Merging [#772][cc-pull] into [development][cc-base-branch] will not change coverage

@@           development       #772   diff @@
=============================================
  Files               17         17
  Lines              531        531
  Methods            103        103
  Messages             0          0
  Branches            67         67
=============================================
  Hits               432        432
  Misses              99         99
  Partials             0          0

Powered by Codecov. Last update [bd4cc96...7f70ca5][cc-compare]
[cc-base-branch]: https://codecov.io/gh/valor-software/ng2-bootstrap/branch/development?src=pr
[cc-compare]: https://codecov.io/gh/valor-software/ng2-bootstrap/compare/bd4cc96cb4d2c0931824eb1a1de66dbdf3e918a6...7f70ca5091179c49cec18c14873846a0f82ff61b?src=pr
[cc-pull]: https://codecov.io/gh/valor-software/ng2-bootstrap/pull/772?src=pr

All good now. Having some issues with the dev environment on windows, so I rely on Travis. Will make some time either to fix it, or create a linux based dev env.

Awesome, thanks!


Use the Application Router in Cloud Foundry to Connect to ABAP System

Tutorials: https://developers.sap.com/tutorials/cp-connectivity-consume-odata-service-approuter.html
Step 5.
The SAP npm registry is deprecated, please remove this part of step 5.
Also, please note that the cockpit UI has changed several times since the screenshots were recorded. It would be great if you could update them as well.
Thanks

Hi @IObert, as already explained by email I am no longer maintaining these tutorials.
I reached out to my manager to find a solution.


[Updated] Added the ability to specify the amount of text to extract and index from an attachment. - On a per call basis.

[Updated] Fixed issue that content length was being set globally on tika instance. It is now setting correctly on a per call basis.
By default, tika is only extracting a maximum of 100,000 characters from the uploaded file attachment. I have modified it so that on upload, you can specify the maximum amount of characters to extract from the document (specify -1 to remove any limit).
Example usage:
{
"my_attachment" : {
"_content_length" : 500000,
"_content_type" : "application/pdf",
"_name" : "resource/name/of/my.pdf",
"content" : "... base64 encoded attachment ..."
}
}

Heya, I will pull this change. I would change teh parameter to be called _indexed_chars, since content length can be confusing. I will also add a generic, index level setting, to control the default value of it (overriding the Tika default one).
I've also opened an issue in Tika to support this in it: https://issues.apache.org/jira/browse/TIKA-870.


Refactor/hooks v1

[!Note]
Review notes at the bottom to get through the bloat of the PR

Refactor

Change all hooks to use wagmi style returns, eg:

const { messages } = useMessages()
//becomes
const { data, isLoading, error } = useMessages()

if(!error && !isLoading) {
  console.log(data.messages)
}

Rename useMessages to useNotifications
Merge useSubscription and useManageSubscription into one hook
Refactor useInitWeb3InboxClient hook into a standard function initWeb3inboxClient. Reduces complexity and is more inline with Web3Modal's API

Cleanup
Remove react components, unused utils and a lot of unused code
Example

Remove unused examples
Use GM hackers as an example

Review Notes
Apologies for the 243 files changed bust most of it removals that can be ignored. Bulk of the changes happens under packages/react/src/hooks and apps/web where the usage of the new hooks can be viewed

Looks great! ✨ I just have a question around GM hackers being moved here. Should we archive the original repo ? Should we also consider moving gm-dapp in here ? It might make your life easier in the long run when you need to update our example dapps.

i think leave gm hackers as a stand alone repo, bundling it in here will make it too cluttered for devs who just wanna find exactly what they need to do to get set up in most minimal way possible. i think putting it in here make it more effort than it needs to find exactly what you need when looking for simple "get started" example


Add minimum Maven version

I've just spent a considerable amount of time chasing after a NPE - turns out since you're using @Inject this plugin requires Maven 3.1.0+, and IntelliJ IDEA's bundled Maven is at version 3.0.5.
It would still bomb out, but with a clearer error.
I could do a PR if you want. Or you could add:
<prerequisites>
    <maven>3.1.0</maven> <!-- or something higher -->
</prerequisites>

to the plugin pom.xml.

Makes sense. I'm only closing as I've moved the issue to JIRA. https://issues.jboss.org/browse/WFMP-25

Yeah, you know, I noticed you moving things to JIRA - is that where I should go ahead and create issues to start, rather than making you do double-entry?

Yeah. I just don't have the permissions to shut the issues off here :)


perf: avoid rehashing block; use stored hash

before 12%

after 10%

Issue being fixed or feature implemented
Redundant rehash
What was done?
Avoid redundant rehash
How Has This Been Tested?
Reindexed 0-500000 on testnet
Breaking Changes
None
Checklist:
Go over all the following points, and put an x in all the boxes that apply.

[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added or updated relevant unit/integration/functional/e2e tests
[ ] I have made corresponding changes to the documentation
[x] I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

Starting a guix build here to ensure that it works here (note this is pre-guix2)


Dataset Viewer issue for gsdf/EasyNegative

Link
https://huggingface.co/datasets/gsdf/EasyNegative
Description
The dataset viewer is not working for dataset gsdf/EasyNegative.
Error details:
Error code:   ClientConnectionError

Thanks for reporting. It sometimes appears, and it's due to a timeout between the Hub backend and the datasets server. We'll be looking at this. Closing in favor of https://github.com/huggingface/datasets-server/issues/877


Add configuration for juju clouds

The runner needs to support a configuration option to configure Juju clouds.
The intention is to enable the use of juju with test runs utilizing https://github.com/charmed-kubernetes/actions-operator to run tests. While the action can configure LXD and MicroK8s based installs, it would also be desirable for a runner to have clouds configured that the tests can use as well. These could allow tests to access pre-conrigured environments like public clouds and private clouds like OpenStack.

If I understand correctly, this can already be done by the user through appropriate workflow configuration, if the infrastructure allows access to the respective cloud. I do not think the charm should expose any other configuration (cc @jdkandersson @yanksyoon @yhaliaw ). Closing the issue now.


Should remove nodejitsu support

Node jitsu was shut down a long time ago. We should remove support for it and remove mentions of it from our README

PR welcome.


Updated URL for the Ondat Azure Marketplace Offer

This is a small PR that updates the Ondat URL to the new Ondat Azure Marketplace offering > https://azuremarketplace.microsoft.com/en-us/marketplace/apps/ondat1652283854761.ondat?tab=Overview

Learn Build status updates of commit 3615fe3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/storage/solution-integration/validated-partners/container-solutions/partner-overview.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@beber-msft

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Learn Build status updates of commit 3e8334e:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/storage/solution-integration/validated-partners/container-solutions/partner-overview.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


Upgrade to Django 1.9.10

Upgrade to Django 1.9.0
I also updated the following libraries for compatibility with this version of Django:
-django-tastypie==0.12.2
+django-tastypie==0.13.0

-django-guardian==1.3.0
+django-guardian==1.4.6

-django-extensions==1.3.8
+django-extensions==1.7.4

-djangorestframework==3.0.4
+djangorestframework==3.1.2

-django-celery==3.1.16
+django-celery==3.1.17

-django-annoying==0.8.4
+django-annoying==0.10.1

# The JSONP serializer was removed from restframework into a new package
+djangorestframework-jsonp==1.0.2

# Uses a more recent version of this unmaintained library, for compat with Django 1.9 and python 3
-git+https://github.com/ericflo/django-pagination.git@e5f669036c#egg=django_pagination-dev
+git+https://github.com/zyga/django-pagination.git@86caf15#egg=django_pagination-dev

@agjohnson I notice there are several views that do not render templates properly. Would it be helpful if there were some tests to check these views and assert the status code is 200? If so, should they live in readthedocs/rtd_tests/tests/test_views.py ?

This requires a rebase and some QA -- we need to make sure this doesn't conflict with anything in our commercial hosting as well. Pending that, we can get this merged. Going to set this as blocked on us for now.

Would it be helpful if there were some tests to check these views and assert the status code is 200? If so, should they live in readthedocs/rtd_tests/tests/test_views.py ?

+1, it would be nice to add the inline test in https://github.com/rtfd/readthedocs.org/pull/2435#issuecomment-251493945.
For those who wonder (like me) why Location headers aren't absolute URLs anymore, please see http://django.readthedocs.io/en/latest/releases/1.9.html#http-redirects-no-longer-forced-to-absolute-uris

I've updated this PR in #2684 to fix conflicts, since it seems like the original PR doesn't allow edits.


wwhrd does not check license of dependencies only used in tests ( *_test.go)

While checking the results of the update in #46, I found out, that _test.go files are skipped. I was not aware of this and I think it is nowhere documented. Anyway, I would like to have an option/flag to enable the inclusion of the _test.go files in order to check all licenses, that are touched by a Go repository.
The respective check is here: https://github.com/frapposelli/wwhrd/blob/dc59affcfaeda9c4b800f9878b652879b4992e04/walker.go#L380

@breml yes, this was a deliberate design choice as testing dependencies are not compiled into the shipping binaries thus normally not subjected to license checks (e.g., I could test my software with a GPL-2 library and distribute it with an incompatible license).
That said, having an optional toggle for that is a good idea. I'll add it to the backlog.

@frapposelli Thanks for your reply and for adding this issue to the backlog. I guess the main problem for me has been, that this behavior is not documented. So as a quick fix, maybe we can just improve the documentation and make it explicit, that dependencies in _test.go are ignored.
I still think, it would be good to have an additional option to control, how dependencies used only in tests are handled. I agree with you, that e.g. GPL-2 is not a problem, when used in test. The other day I realized though, that we have used https://github.com/bouk/monkey in one of our tests and the license is pretty adventurous from a business point of view, even if used in tests.
With all of this said, I think it would be good to have the flag to check the licenses used in tests as well. Additionally I assume, that this change would make it necessary to have the respective sections in the config file as well to config a blacklist, whitelist and exceptions for licenses used only in tests.

Cutting v0.4.0-rc.1 with this enhancement right now.


Multiple concurrent Process Monitor processes sometimes fails to finish

I wrote a small program in pony to run a few (4 to be exact) processes in parallel, parse the output, etc, and noticed that sometimes my program hung while one of the subprocesses was stuck in the zombie
state.  The zombie process corresponded to the ProcessNotifier that didn’t have dispose called, so my code was hung waiting for dispose. The code below reliably triggers the problem on my system here (Linux x86_64, ponyc 0.21.3, both built from git & nix). The sample program starts a number of sleep 0.1 processes in parallel; the single command line argument is the number of subprocesses to create. Testing like this:
for i in $(seq 0 1000000); do echo -n $i: ; ./result/bin/process_test 1; done

runs perfectly with 1 test process, so the problem does not seem to ever trigger with a single subprocess:
999994:Started processes: All finished
999995:Started processes: All finished
999996:Started processes: All finished
999997:Started processes: All finished
999998:Started processes: All finished
999999:Started processes: All finished
1000000:Started processes: All finished

With 2 processes it generally takes somewhere between 500 to 1500 iterations for the program to hang, 3-4 < 50, and with > 10 subprocesses it generally hangs each time.  Note that with > 10 subprocesses there is often more than one process stuck in the zombie state.
use "files"
use "process"

actor Main
  let _env: Env
  var _num_running: U64

  new create(env: Env) =>
    _env = env
    _num_running = 0
    try
      let auth = _env.root as AmbientAuth
      let path = FilePath(auth, "/usr/bin/sleep")?
      var i: U64 = 0
      while i < _env.args(1)?.u64()? do
        start_process(auth, path)
        i = i + 1
      end
      _num_running = i
      _env.out.write("Started processes: ")
    else
      _env.out.print("Number of processes to start is a required argument!")
    end

  fun start_process(auth: AmbientAuth, path: FilePath) =>
    let args = recover Array[String](3) end
    args.push("sleep")
    args.push("0.01")
    let pm = ProcessMonitor(auth, auth, ProcessClient(_env, this), path, consume args, _env.vars())
    pm.done_writing()

  be finished_ok() =>
    if _num_running > 0 then
      _num_running = _num_running - 1
      if _num_running == 0 then
        _env.out.print("All finished")
      end
    end

class ProcessClient is ProcessNotify
  let _env: Env
  let _main: Main

  new iso create(env: Env, main: Main) =>
    _env = env
    _main = main

  fun ref stdout(process: ProcessMonitor, data: Array[U8] iso) =>
    None

  fun ref stderr(process: ProcessMonitor, data: Array[U8] iso) =>
    None

  fun ref failed(process: ProcessMonitor, err: ProcessError) =>
    _env.out.print("ProcessError!")

  fun ref dispose(process: ProcessMonitor, child_exit_code: I32) =>
    if child_exit_code != 0 then
      _env.out.print("Child exit code: " + child_exit_code.string())
    end
    _main.finished_ok()

I believe I have found the actual issue; in order to find out what was actually going on, I added ~50 trace/prints through the ProcessMonitor, asio/event and then asio/poll until the difference between processes that exited correctly and processes that did not became clear. This is a process that finishes correctly (greping output for fd 11):

After a session of printf debugging the problem appears to be clear enough, here is the output of my customised pony runtime / process_monitor (lots of
pony asio event create fd 11 ev 0x7f50d54159c0 read 1 write 0
subscribe epoll fd 11 ev 0x7f50d54159c0 is noisy, old_count: 10
3: _try_shutdown: not ready to close: stdin fd<PHONE_NUMBER> stdout fd 11 stderr fd 16
got event for fd 11 ev 0x7f50d54159c0
got read event for fd 11 ev 0x7f50d54159c0
poll sending event for fd 11, ev 0x7f50d54159c0 auto_resub: 0, read: 1, write: 0
3: enter: _pending_reads fd 11 status: stdin fd<PHONE_NUMBER> stdout fd 11 stderr fd 16
unsubscribe epoll fd 11 ev 0x7f50d54159c0 is noisy, old_count: 8
pony asio event destroy fd 11 ev 0x7f50d54159c0

sorry for accidentally closing the issue, basically I have figured out the root cause, so the problem is that asio subscribe/unsubscribe requests to the same fd overlap, so in a ponyc with lots of extra printfs this is the normal lifetime of an fd of a subprocess:
pony asio event create fd 11 ev 0x7f50d54159c0 read 1 write 0
epoll subscribe fd 11 ev 0x7f50d54159c0
epoll got read event for fd 11 ev 0x7f50d54159c0
epoll sending event for fd 11, ev 0x7f50d54159c0 read 1 write 0
epoll unsubscribe fd 11 ev 0x7f50d54159c0
pony asio event destroy fd 11 ev 0x7f50d54159c0

but for the problematic fd that is stuck this happens:
pony asio event create fd 12 ev 0x7f50d5423d80 read 0 write 2
epoll subscribe fd 12 ev 0x7f50d5423d80
epoll got write event for fd 12 ev 0x7f50d5423d80
epoll sending event for fd 12, ev 0x7f50d5423d80 read 0 write 2
pony asio event create fd 12 ev 0x7f50d5415800 read 1 write 0
subscribe epoll fd 12 ev 0x7f50d5415800
unsubscribe epoll fd 12 ev 0x7f50d5423d80
pony asio event destroy fd 12 ev 0x7f50d5423d80

Initially it is subscribed for writing, an event arrives and then another new asio event is created for the same fd for reading instead of writing. However, the unsubscribe from the write event is happening after the subscribe for the read event, epoll is removing the fd from the epoll set and of course asio never delivers any more notifications for that fd; obviously my test program then hangs at that point.
are also some other oddities in the debug output I've generated wrt the asio noisy counter, it  to be incremented , I'll try to tidy up my printf'd code tomorrow and push it for reference.

A followup: the basic problem is that in the process monitor code, the fds are being closed as the close events arrive from asio, but the unsubscribe only happens later in _close. There is a hard rule here: the unsubscribe must happen before closing the fd, or else there is a chance for the closed fd to be reused with a different subscribe. Making this change in the ProcessMonitor makes everything run almost perfectly, I have however had the program above (with 10 subprocesses) fail to exit more than once; as far as I could tell the asio thread was not being set "unnoisy" in pony_asio_event_unsubscribe, the subprocesses had all been reaped, but for some reason asio was still active. I'm currently trying to reproduce it here, could take a while.

I started writing a PR for the commit above, but then the hang triggered again after 30k runs; this is different, subprocesses have been reaped, but the process is still running. There are still two read pipes open.

I wasn't aware of #2561, but it does look very similar to the remaining hangs, iirc I was seeing a situation where asio had sent an event for a ProcessMonitor actor, but the ProcessMonitor actor was not waking up. I just rebuilt current ponyc master fc6a6048bc4c47772a553db9a96f7528c55f9909 and the test program from #2529 is currently burning a hole in my computer, currently at 7k iterations and climbing.

270k iterations and still going; I'll leave it running overnight, but it looks like that as of fc6a604 this is fixed and can be closed.

Successfully ran the test program overnight on current master fc6a604, a million iterations with 20 subprocesses; originally this would hang on almost every iteration, with #2574 it takes several thousand iterations to hang, adding #2561 seems to make everything work perfectly.

This is great news! Thanks for the time, brain and CPU cycles spent on that!


Cannot find Eigen/Sparse

After successfully building Caffe, correctly configurating CUDA versions (see #15 if you have trouble), and making sure CommonSettings.props is copied over from Caffe, the build still fails.
Could find Eigen/Sparse, referenced in WLS.cpp.

I was able to resolve this issue as follows:

Download Eigen from here (.tar.gz, .zip, whichever suits you best).

I used the latest version, which is currently 3.3.4.

Extract the file.  You will have a folder similar to eigen-eigen-5a0156e40feb

Depending how you extract it, you might have an eigen-eigen-5a0156e40feb within another eigen-eigen-5a0156e40feb.  You want the child-most one.  It should have folders like bench, blas, cmake, etc, inside of it.  Visual Studio will need to be able to find eigen-eigen-5a0156e40feb\Eigen later on.

Presumably by this point you have already added deep_image_analogy.vcxproj as a project to your Caffe solution.  If not, go here and figure out how to build Caffe first (within VS 2013 at the time of writing, though in the near future 2017 might get supported).
Right click your deep_image_analogy project in the solution explorer within your Caffe solution and click "properties".
Configuration Properties > C/C++ > General, then edit Additional Include Directories to include the absolute path to your unzipped eigen-eigen-5a0156e40feb

This should resolve the dependency issue.  Eigen is only a template solution, there aren't any headers you should have to prebuild or anything.


Collection: pass down intensity

As of now, the intensity of a collection is not passed down to it's member functions.

I don't know which is the desired behavior.

It seems the expected behaviour to me but it's incomplete.
This is OK to set the startup intensity of the children functions, but the interesting case is when a Collection is a Chaser step, with fade in and fade out.
I guess that's what users are asking since quite a while now

Alright, I'll complete this later :)


fix: api decorator value not set correctly

The previous value will be the same for all instances of ApiStream which will prevent using different access tokens for other ApiStream instances

@letranloc Thanks for the contribution! This fix just got released as v1.0.22 👍🏻


Look into using MATCH to improve the labeler

https://github.com/yuzhimanhua/MATCH
https://arxiv.org/pdf/2102.07349.pdf

Subtask: https://github.com/nasa-petal/PeTaL-labeller/issues/43

We'll have to verify that adding author and venue metadata helps (not hurts) performance.
MATCH uses citations, venue, and authors as metadata.
SPECTER only uses citations. Here's their explanation:

More surprisingly, adding authors as an
input (along with title and abstract) hurts performance.
One possible explanation is that author
names are sparse in the corpus, making it difficult
for the model to infer document-level relatedness
from them. As another possible reason of this behavior,
tokenization using Wordpieces might be
suboptimal for author names. Many author names
are out-of-vocabulary for SciBERT and thus, they
might be split into sub-words and shared across
names that are not semantically related, leading
to noisy correlation. Finally, we find that adding
venues slightly decreases performance, except
on document classification (which makes sense, as
we would expect venues to have high correlation with paper topics).

Log metrics such as precision, recall, f-score, and confusion matrix, to identify the effect of different algorithms, parameters, datasets, and metadata.

Okay, will look into this! This is new to me, so what else do we know already about MATCH vs. SPECTER vs. other approaches?

@dsmith111 finished reformatting our labelled dataset for MATCH training/testing. JSON file is here: https://github.com/nasa-petal/PeTaL-labeller/blob/main/scripts/lens-cleaner/cleaned_lens_output.json


PR #408 Breaks Highlight.js

The change to the css added in #408 is more specific than the css used by highlight.js to syntax highlight code. Pages that were previously syntax highlighing fine don't once casper is upgraded to the latest version.
Highlight.js just uses spans for coloring, e.g. def. If the selector is limited to elements that were displaying invisibly it should fix the issue. I'll try to submit a PR once I get chance, but it'll probably be later in the week. In the mean time if anyone has a better suggestion on how to fix it, or if you'd rather not fix the problem here since it's due to third-party css, let me know.
Steps to Reproduce

add <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/obsidian.min.css"> to the page header
add the following to page footer:

add a code snippet to the page

    ```scala
    def aScalaFunction(): Unit
    ```

You should see syntax highlighted text, instead it is just white on black.
Technical details

Casper Version: 2.1.7
Ghost Version: 1.18.4

if you'd rather not fix the problem here since it's due to third-party css

Yeah, pretty much? Not sure what else to say about this one. If you load a 3rd party script/style which breaks things, then that's generally a problem to report to the 3rd party script/style

The problem is not due to third party CSS. The third party CSS is fine and does not break things. The change introduced in PR #408 just "hides" all colors that are added by the syntax highlighter (highlight.js).

Highlight.js is not part of casper and has nothing to do with casper. If you introduce a 3rd party library, then it's up to you to deal with compatibility. It is not up to Casper to anticipate every 3rd party library in the world which may or may not be added at some point. If you introduce a new displaynone.css which sets body { display: none; } then equally - that is not a bug in Casper.
If Casper doesn't work for your particular use-case, then download it and modify it for your needs. That's how open source works.


Only use timesteps where Sun's elevation is > 10 degrees above horizon

Throw away timesteps where, for a given geo region of interest, any geo location in that ROI has a Sun elevation <= 10 degrees.

Im not sure this is critical - We can still train on data where <10, but just the outcome metrics shouldnt include these

Yeah, I agree it's not a total show stopper if we don't do this. But it will probably improve our ML performance if we ensure the train set has the same statistical properties as the test set. And this should be fairly easy to implement, I'd guess, because you've already done the hard work of computing the elevation :)

I thought would easily be able to do this in Sun.datetime_index
But infact its a bit tricky as Sun.elevation is a dataframe of time vs (x,y)-location. This means the datetimes are the location dependent, but that goes a bit against our manager.py workflow - as 'get locations' is from just one 'data_source'
Perhaps there is an easier way ....
Next step:
plot elevations throughout the year across the UK - for understanding of things

So potentially we just remove anytimes stamps  where elevation <10 for Shetlands - this means we might lose some data where the elevation is >10 int he south but not in the north,
We will only really loss timestamps in winter though

Another way could be to edit [select_daylight_datetimes](https://github.com/openclimatefix/nowcasting_dataset/blob/main/nowcasting_dataset/time.py#L20) to select elevation > 10

Perhaps the easiest to do is, only select datetimes where 'london' > 10 for training, and then in the evaluations metrics we are a lot stricter with the filtering.

Perhaps the easiest to do is, only select datetimes where 'london' > 10

Yeah, I agree, let's just use that criterion for now!  Simple is good :slightly_smiling_face: (simple is good for our code; and simple is good in terms of explaining our code to ESO!)  TBH, I would guess the "London" criterion is probably OK for testing too (at least, for WP1), because most of Britain's PV capacity is in the South.  What do you think, @peterdudfield & @jacobbieker?
Then, in early 2022, as you say, we can think about how to re-architect Manager so that, when it generates the list of locations for each example, it takes into consideration the issue you've raised above:  that the Sun's elevation is dependent on the location and the datetime.
How does that sound?

We should ask Dan too... I'll add this issue to today's "OCF Sync" meeting so we can get Dan's opinion...

Great plots!

Perhaps the easiest to do is, only select datetimes where 'london' > 10

Yeah, I agree, let's just use that criterion for now! Simple is good 🙂 (simple is good for our code; and simple is good in terms of explaining our code to ESO!) TBH, I would guess the "London" criterion is probably OK for testing too (at least, for WP1), because most of Britain's PV capacity is in the South. What do you think, @peterdudfield & @jacobbieker?
Then, in early 2022, as you say, we can think about how to re-architect Manager so that, when it generates the list of locations for each example, it takes into consideration the issue you've raised above: that the Sun's elevation is dependent on the location and the datetime.
How does that sound?

I think thats very sensible

Then, in early 2022, as you say, we can think about how to re-architect Manager

Here's a new issue to remind us to do this re-architecting in 2022: #421

Yeah, I think that sounds good too!

This will be fixed when PR #417 is merged


*: add an unsupported optimistic shard DDL statement

What is changed, added or deleted? (Required)
add an unsupported optimistic shard DDL statement
Which DM version(s) do your changes apply to? (Required)

[x] master (the latest development version, including v3.0 changes)
[x] v2.0 (TiDB DM 2.0 versions)
[ ] v1.0 (TiDB DM 1.0 versions)

What is the related PR or file link(s)?

This PR is translated from:
Other reference link(s):

Do your changes match any of the following descriptions?

[ ] Delete files
[ ] Change aliases
[ ] Have version specific changes
[ ] Might cause conflicts after cherry-picked

@lance6716 @CharLotteiu PTAL

LGTM

LGTM. @yikeke PTAL.

/merge


GROOVY-9677: provide a library of pre-canned macro methods

Could it be named simply "groovy-macros"?

could be but I think that is too close to groovy-macro

Merged. Thanks.


Printing to console an output fails my creation script

When creating, a new resource via market space script (python).
Error message [ErrorDetail(string='Unexpected structure of output', code=['mattronix,', 'has', 'been', 'deployed.'])] Error traceback Traceback (most recent call last): File "/usr/src/waldur/src/waldur_mastermind/marketplace/utils.py", line 104, in process_order processor(order).process_order(user) File "/usr/src/waldur/src/waldur_mastermind/marketplace/processors.py", line 70, in process_order scope = self.send_request(user) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/waldur/src/waldur_mastermind/marketplace_script/processors.py", line 73, in send_request raise serializers.ValidationError( rest_framework.exceptions.ValidationError: [ErrorDetail(string='Unexpected structure of output', code=['mattronix,', 'has', 'been', 'deployed.'])]
Here is the script for example:  https://github.com/mattronix/waldur-custom-offerings/blob/main/create_user_over_ssh/storage-user-add-waldur.py

yes, this is intended
this will be documented for https://github.com/waldur/waldur-mastermind/issues/50 .
Meanwhile you can check the code -
https://github.com/waldur/waldur-mastermind/blob/develop/src/waldur_mastermind/marketplace_script/processors.py#L48
Basically, creation script expect either a single word in the last line of the output, or 2 words with second being base64 encoded metadata.

what is the backend_id and metadata used for?

Dont quite follow what the backend_id is for can you help me in the right direction?
Do you maybe have a slack or chat group for this kind of stuff

Hi,
first of all, sorry for incorrect estimations as to when we can get to the issues you've raised. As they say, if you want to make God laugh, tell him your plans. Anyways, we don't have really chats/groups for non-paying customers, but send me some contact details to<EMAIL_ADDRESS>with messenger details of your choice (signal, slack or skype are ok) and I will add you.
On the topic - backend_id is used for referencing a unique 'isolation' in the backend system matching resource in waldur.
E.g. openstack tenant uuid for a resource matching tenant, or a slurm hpc allocation name for slurm.

No problem, dont want to be a bother or anything just checking in from time to time :)
Il send you my contacts since probably most of my questions are not really good for issues ahahahha.
Thanks anyway for all the help so far :)

Added documentation on output formats: https://docs.waldur.com/admin-guide/providers/custom-scripts/#script-output-format


stroke animation Help: Starts at wrong Direction

https://codepen.io/thapliyalshivam/pen/pdQaeg
I am trying to animate merry to draw from M to Y. It starts with Y to m which isn't what I am trying to do. I trying offset but because of Bounce thing, it doesn't look good as a few part of the stroke still is visible at the ends. Is there any value flag?

The SVG path of merry is exported in the wrong direction. You can easily change the direction of your SVG path by installing this script in Illustrator. If you want to change the easing of the animation please check out the documentation.


Add Contributors Table in README.md

Added contributors table in README.md file
Related Issuse or bug
Fixes: #25
Describe the changes you've made
I have made README.md file beautiful and given credit to contributors
Describe if there is any unusual behaviour of your code(Write NA if there isn't)
NA
Checklist:

[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[ ] I have added tests that prove my fix is effective or that my feature works.
[ ] New and existing unit tests pass locally with my changes.
[ ] Any dependent changes have been merged and published in downstream modules.

Screenshots
Original

Updated

Nicely done!

Nicely done!

@Femme-js Kindly add "-" in level label e.g. "Level-2"

@Femme-js Kindly add "-" in level label e.g. "Level-2"

@Femme-js Kindly add "-" in level label e.g. "Level-2"


Change maintainer to Github user instead of organisation

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

noarch: python recipes are recommended to have a lower bound on the python version. This recommendation will become a requirement in the future.

@conda-forge-admin, please rerender

@conda-forge-admin, please ping core team
Please merge this PR to give maintainer rights to my personal GitHub user.

Hi! This is the friendly automated conda-forge-webservice.
I was asked to ping @conda-forge/core and so here I am doing that.

@FuzesiMate -- could you

increment the build number from 0 to 1
Under requirements - run and host change - python to - python =2.7|>=3.4 to satisfy the linter and match what is in the setup.py.

Then I'd be happy to merge.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@FuzesiMate -- could you

increment the build number from 0 to 1
Under requirements - run and host change - python to - python =2.7|>=3.4 to satisfy the linter and match what is in the setup.py.

Then I'd be happy to merge.

@synapticarbors Thank you for your help! I have increased the build number and added lower bound to Python version.


Jerrymathew somewhatcool new updates

adding a gif for virtual graduations

@jerry-mathew - it looks like the gif you reference might not have gotten committed.
Can you check if this file is available to commit and push? resources/virtual-grad.gif

Great addition! Thanks!


Evaluate database(Cosmos, Azure SQL etc) security of all templates

I would like to mention in this context that https://github.com/Azure-Samples/todo-csharp-sql effectively disables the SQL firewall by adding an allow-all rule, which is not mentioned anywhere in the readme. Given that this is going to be taken as a reference solution I'd argue this is effectively putting a bomb in code that people will trust is aligned to best practices which it very much is not.

Also consider adding how the password generation works in the security section. https://github.com/Azure/azure-dev/issues/994 has the source code. From what I'm reading, azd calls this command internally, always expects a kv name and key, looks up the secret using the key provided from kv, if it exists, returns it, else generates a new one and returns that

Not a must-fix for GA. Moving to Germanium.
Since Bicep related efforts i.e, moving bicep modules(azure-dev -> awesome-azd), following best practices, moving into bicep registry will be handled in Germanium.
@savannahostrowski @ellismg


CloudFormation: when no tags are present, Yor renders an incorrect template

Take this sample minimal S3 CloudFormation input template:

---
AWSTemplateFormatVersion: 2010-09-09
Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: PublicRead
      WebsiteConfiguration:
        IndexDocument: index.html
        ErrorDocument: error.html
    DeletionPolicy: Retain

when running yor tag -d ., yor adds the required tags, but doesn't add a 'Tags:' directive:
---
AWSTemplateFormatVersion: 2010-09-09
Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: PublicRead
      WebsiteConfiguration:
        IndexDocument: index.html
        ErrorDocument: error.html
    DeletionPolicy: Retain
    - Key: git_repo
      Value: yor-sandbox
    - Key: git_file
      Value: s3.yml
[...]

thus rendering the CloudFormation template invalid.
I would expect yor to issue an error saying it expects the 'Tags:' directive, or to add it itself.
When tagging a similar minimal Terraform s3.tf file, yor behaves correctly.

Thanks for reporting @kristofwillen !