Add generator for bootstrap4

Bootstrap 4 will be there soon and some form related things changes.
For a list of changes see http://v4-alpha.getbootstrap.com/migration/#forms
Please add a new generator for bootstrap4.
Maybe as a new option rails generate simple_form:install --bootstrap4? Also a note/warning at the end of the default "--bootstrap" call would be got, something like "You just generated the form styles for bootstrap3. Should you require bootstrap4 styles, re-run the generator with rails "generate simple_form:install --bootstrap4 --force"."
Update:
I started the work over here, please feel free to pick it up https://github.com/plataformatec/simple_form/pull/1337

There is new information about one of the open details at https://github.com/twbs/bootstrap/issues/18705#issuecomment-168248128

class="form-group" should be changed to class="form-group row"

Is this going to be added as an option? I would like to see this made as Bootstrap 4 is very nice with flexbox additions.

As bootstrap 4 is officially released would seem like a good time to update bootstap generator?

+1 Bootstrap 4 official version is now released.
Are there plans to support this or has anyone done this already? New to simple_forms as been using formtastic, but a bit surprised that BS4 isn't supported already. I could potentially do this but looking at previous comments it looks like some changes to simple_form internals are required to support the changes...

any updates on this?

This would be great to have bootstrap 4 support. Booleans currently don't render correctly.

The direction we intend to follow is to decouple simple form from any CSS framework, so that specific gems can be created to support them better.
Please take a look at https://github.com/plataformatec/simple_form/pull/1337#issuecomment-350709787

Closing it since we won't add support to Bootstrap 4.
We are working on simple_form-boostrap to update to Bootstrap 4 and making Simple Form flexible to support it. Feel free to join to us.
Thanks

@feliperenan its great to see progress in this other repo. However, I still think this Repo should have an open issue that people can find and understand whats going on. This repo still needs to be updates about bootstrap 4, eg:

Update readme https://github.com/plataformatec/simple_form#bootstrap about Bootstrap 4.
Make it clear somewhere, what the new policy about code changes to simple_form for other frameworks is.
Remove or rename the 'bootstrap' generator which is misleading.

Also, https://github.com/rafaelfranca/simple_form-bootstrap/pull/56 seems to be depending on some special version of simple form (see https://github.com/rafaelfranca/simple_form-bootstrap/pull/56/files#diff-8b7db4d5cc4b8f6dc8feb7030baa2478R72). So are there plans now to merge external changes into simple_form? In one of those open tickets the questions was answered with 'no'.

Also, rafaelfranca/simple_form-bootstrap#56 seems to be depending on some special version of simple form (see https://github.com/rafaelfranca/simple_form-bootstrap/pull/56/files#diff-8b7db4d5cc4b8f6dc8feb7030baa2478R72).

This is just temporary and for demo purposes, including a feature request (https://github.com/plataformatec/simple_form/pull/1559) about having valid_class on input elements also which now has it's own PR.

Looking forward to see a PR from you. Open source also means reacting and not complaining. ✌

Excuse me, gentlemen, for commenting in a closed topic but I (and I believe many other users) will find this situation highly confusing. What is the best course of action here? I use Bootstrap 4 in my app. This will not change. Can I use just helper methods from simple_form (e.g. simple_form_for) to leverage simplified field generation but wrap the generated code in my manually written Bootstrap 4 code in views - that means no initializers?
Sorry for asking probably an obvious question but I do not see any straightforward answer. Thank you.

@tordans  For me, the issue (Add generator for Bootstrap 4) was resolved so we can keep it closed.
I agree with you, we have some work to do about Bootstrap and I include the Foundation as well. IMHO we can remove the initializers and definitely remove the support for it, but I want to see other opinions about it. I will open another issue explaining the motivations and we can discuss it there.
@technodrome  You can use Simple Form with Bootstrap 4 but some things maybe will not work, we are working on it - making Simple Form flexible enough to work with any CSS framework. When we talk about don't support Bootstrap anymore, we are talking about Simple Form generating the configuration out-of-the-box.


MT GC

This PR allows the GC to work with threads running different fibers. It does not change the scheduling at all. There is a sample program added that manually schedule fibers to threads.
To test this branch you need a custom build of the bdwgc.
The path to find the libgc.a is fixed for now.
To be able to run this you need checkout and build in a specific location:
In your crystal working directory with mt/gc branch checked out:
$ pushd ..
$ git clone http://github.com/crystal-lang/bdwgc
$ cd bdwgc
$ git checkout release-7_6-crystal
$ git clone https://github.com/ivmai/libatomic_ops
$ cd libatomic_ops
$ git checkout v7.6.10
$ cd ..
$ autoreconf -vif
$ ./configure --disable-debug --disable-dependency-tracking --disable-shared --enable-large-config
$ make
$ ls .libs/
libcord.a   libcord.la  libcord.lai libgc.a     libgc.la    libgc.lai

$ popd # Go back to the crystal working directory

$ make clean deps

# To run the samples/mt_gc_test.cr (no need to build the compiler)
$ ./bin/crystal samples/mt_gc_test.cr -D preview_mt_gc

# To build the compiler itself
$ make crystal FLAGS="-D preview_mt_gc"

The PR adds a simple flag to check whether a fiber is running or not. This is only used for GC purposes and the GC disable/enable in the context switch ensures the information is only read in a consistent state. Yet that check might be affected by the work in #7407
One of the extensions of bdwgc GC_get_stackbottom is to get the stackbottom of the current thread.  That might not be needed if #6970 evolves successfully.
The other extension of the bdwgc GC_switch_to_corutine and how it is used in the Scheduler/Fiber are the real game changer here.
The whole patch to bdwgc can be found at https://github.com/crystal-lang/bdwgc/commit/417756b3b6f00f08b9d92f8dcd51c539402a3403

This is based on some of the past efforts of (and present talks with 😄) @ggiraldez and @waj

So, the Thread.new method has already been available before, but new threads were not GC'ed, right? And if this got merged, then we can create true multi-threaded applications?

Do you get acceptable performance?
I initially used GC.disable and GC.enable while running a context switch, but the performance was awful. I also tried masking the signal used to stop-the-world, which improved performance to not that horrible with multiple threads (but very bad with a single one), but it was linux only, so not portable. See https://github.com/crystal-lang/crystal/pull/7214#issuecomment-450115098 for benchmarks.
I eventually did what the single-msqueue branch did: use a rwlock and performance got surprisingly acceptable (not using ConcurrencyKit but a custom multi-reader single-writer lock). See https://github.com/ysbaddaden/crystal/commit/ab7bfb4431f6eecaa26ef982b813936cd7fca587 and a benchmark comparable to the previous disable/sigmask link (1, 2, or 4 threads switching between 128 fibers):
switch[1/128]: crystal: 10000000 yields in 411 ms, 24285940 yields per second
switch[2/128]: crystal: 10000000 yields in 991 ms, 10082371 yields per second
switch[4/128]: crystal: 10000000 yields in 949 ms, 10526425 yields per second

I also push thread stacks on GC collect, not on each and every context switch. I never benchmarked, but since there are only a few threads, but I assume it's faster.

@ysbaddaden currently just using the std spec, compiler specs and the sample. I didn't experience slowdown but probably because the only MT code checked is the sample (and is unable to run in ST).
The GC.disable/GC.enables could be wrapped by flag?(:preview_mt_gc) if needed to allow checking things in MT before enabling the feature.
I try to avoid calling GC_switch_to_corutine (aka set_thread_stack_bottom) in the context switch and calling it just in GC.before_collect but that didn't work. Some collections can be triggered by the gc after a context switch but did not trigger the before_collect callback and then the stack information in the GC was wrong causing a lovely segfault. Having a set_thread_stack_bottom(thread, stack_bottom) would be better in terms of performance but I didn't succeed. So I ended up using a more declarative API like GC_switch_to_corutine. The sample script with 8 threads and 100 fibers did consistently trigger a collection with wrong stack information for one of the threads.

Would be better to wrap then GC.disable/GC.enables to at least avoid hitting down the ST performance for now, right?

Should I use :mt or :preview_mt as a flag directly so other MT features can be enabled all together?

From https://github.com/ysbaddaden/crystal/commit/ab7bfb4431f6eecaa26ef982b813936cd7fca587 , we could include the RWLock and add the integration with boehm. I am not sure if it's sound to do the change on atomic.cr from :sequentially_consistent to :acquire/:release. It seems the former puts stronger constraints

Sequentially-consistent ordering
Atomic operations tagged memory_order_seq_cst not only order memory the same way as release/acquire ordering (everything that happened-before a store in one thread becomes a visible side effect in the thread that did a load), but also establish a single total modification order of all atomic operations that are so tagged.

I will submit another set of instructions that will require creating a .pc file ¬¬ and remove the hack path so we can have better chances on merging and testing this.

I would like to have some specific benchmark ready to run and try to decide better if the rwlock is worth it. Do you have the http stress test that you used?

@vladfaust Thread.new was already there, not as a public api. As long as you don't use fibers/io on secondary threads everything is fine. So, don't use it 😸.

Minor, but is corutine a typo (expected coroutine), or was that intentional for name collision reasons?

I need to learn english. Thanks @Fryguy 😸

It took me some time to understand the atomic memory ordering constraints, but AFAIK the memory ordering is declaring memory barriers for LLVM optimizer's to not reorder across atomic instructions. Since I only use Atomic::Flag for creating locks, I only care for instructions within :acquire and :release to never be moved before or after the atomic instructions, but instructions outside of the atomics can be reordered within. :seq_cst is more strict, and nothing can cross the memory barriers.
I believe that acquire/release is fine for locks. I'm not sure there are other use cases for Atomic::Flag that would need a strict sequential memory ordering?

I've rebased this on (almost) master and integrate changes from #7551, https://github.com/crystal-lang/crystal/pull/7551#discussion_r266594361, and compare with the GC aspects of #7547.
#7547 helped me identify the problem mention at https://github.com/crystal-lang/crystal/pull/7546#issuecomment-473021383, I missed swapping the procs call in GC.before_collect.
I've created a couple of branches to compare strategies (more on that later).
To check this you first need to build GC with custom extensions. It match this patch.
$ pushd ..
$ git clone http://github.com/crystal-lang/bdwgc
$ cd bdwgc
$ git checkout mt-gc-benchmark
$ git clone https://github.com/ivmai/libatomic_ops
$ cd libatomic_ops
$ git checkout v7.6.10
$ cd ..
$ autoreconf -vif
$ ./configure --disable-debug --disable-dependency-tracking --disable-shared --enable-large-config
$ make
$ ls .libs/
libcord.a   libcord.la  libcord.lai libgc.a     libgc.la    libgc.lai

Also, create the following gc.pc file, changing the /path/to/bdwgc with the full path of the bdwgc working directory.
$ cat gc.pc
prefix=/path/to/bdwgc
libdir=${prefix}/.libs

Name: bdwgc
Description:
Version: 7.6.12+mt
Libs: ${libdir}/libgc.a

Then we build the samples/mt_gc_test.cr. It now has some command line options to measure its performance.
I've have checked some alternatives. Each of it in their own branch.

branch: mt/gc-switch_to_coroutine. The GC_switch_to_coroutine is used in Fiber.resume. GC_enable/GC_disable for locking.
branch: mt/gc-set_stackbottom-in-resume-lock. The GC_set_stackbottom is used in Fiber.resume with Thread.current. GC_enable/GC_disable for locking.
branch: mt/gc-set_stackbottom-lock. The GC_set_stackbottom is used in GC.before_collect. GC_enable/GC_disable for locking.
branch: mt/gc-set_stackbottom-rwlock. The GC_set_stackbottom is used in GC.before_collect. Custom GC::RWLock for locking.

Note: these branches are in my fork

The last three uses the same GC extension. So I wanted to check if there was any benefit of using the implicit current thread or not.
If not is better to have a more flexible API since it can be used, as shown, in different ways (Fiber.resume vs GC.before_collect).
After checking out each STRATEGY branch, to build the mt_gc_test the following can be used:
$ rm -rf ~/.cache/crystal/ # clear temp files because reasons
$ (export PKG_CONFIG_PATH=/path/to/bdwgc:$PKG_CONFIG_PATH && ./bin/crystal build samples/mt_gc_test.cr --release -D preview_mt -o gc-STRATEGY)

Note: I changed the flag to preview_mt.

Additionally as a baseline a -D gc_none can also be compiled:
$ rm -rf ~/.cache/crystal/ # clear temp files because reasons
$ ./bin/crystal build samples/mt_gc_test.cr --release -D gc_none -o gc-none

Finally, things like the following can be used to measure the performance of different strategies.
$ ./gc-none -m -f 1000 -t 2 -l 20
$ ./gc-switch_to_coroutine -m -f 1000 -t 2 -l 20
$ ./gc-set_stackbottom-in-resume-lock -m -f 1000 -t 2 -l 20
$ ./gc-set_stackbottom-lock -m -f 1000 -t 2 -l 20
$ ./gc-set_stackbottom-rwlock -m -f 1000 -t 2 -l 20

I've tested this on OSX for a couple of variables. Results can be viewed here.
So far it seems that the best alternative is gc-set_stackbottom-in-resume-lock, but they are all very similar.
The gc-set_stackbottom-rwlock seems to hang on some examples, but if that strategy ends up working the same GC extension could be used.
I have moved the mt/gc branch to mt/gc-set_stackbottom-in-resume-lock so that can be reviewed/approved in this PR.
Of course, if more experiments shows that another alternative is better we can switch that. But for now, that strategy seems enough.

I did the edits in the samples code that @ysbaddaden the most notable changes is that the gc_none version is even faster. The others didn't change much for now. So I will stick with the previous decision of using mt/gc-set_stackbottom-in-resume-lock as a starting point.

I tweaked my implementation a bit to try different lock strategies. See https://github.com/ysbaddaden/crystal/commit/093060042a149c777fee81678daae4e3a62dfeaa and https://github.com/ysbaddaden/crystal/commit/58af519543a56040eb59e950d8e325064a828a32.
Using my HTTP::Server with different tests (IO bound, GC bound, CPU bound) I got slightly better values with pthread_rwlock and my custom rwlock than with GC.disable but nothing really significant. I think the overhead mostly lies in libevent and IO syscalls and somewhat erases the context switch overhead.
Yet, using my context switch benchmarks, the overhead of a context switch (including the resumable CAS 1 to 2) is significantly different using the different strategies. See this plot, using gc_none as a reference (zero overhead):

I'm worried about the hangup you got with my rwlock, but I couldn't reproduce it, despite running HTTP::Server with a GC collect running every some requests :/

I'm fine with using GC.disable for now, but I'd love the GC to have the 4 lock_read, lock_write, unlock_read and unlock_write methods, so we could change what's used.

I'm so happy to see the 0.28 milestone on it! :rocket:
Drum beat intensifies :drum:

@ysbaddaden with the last push

The GC has the 4 lock_read, lock_write, unlock_read and unlock_write methods, yet only one strategy is used.
Changing the strategy should imply changes only in boehm.cr
I fixed the hang problems. There was sync issue with the worker threads when finishing/starting each round of the stress test that is fixed in https://github.com/crystal-lang/crystal/pull/7546/commits/81783cfd7b1ceb5b34570a3686ab232ded4bf53f.
Thread.current is used several times already in a context switch: to access the scheduler, to set the stack_bottom. Accessing it one more time to check the single resume isn't problematic. I still prefer to have the Fiber::Context as an @[Extern]

I don't have time to review properly. From your last comment, the patch sounds acceptable.
About Thread.current there should ideally be one single call to it during a context switch: to reach the current thread's scheduler. Setting the stack bottom can be moved to GC.before_collect without delaying GC marker threads (see https://github.com/crystal-lang/crystal/pull/7547/commits/093060042a149c777fee81678daae4e3a62dfeaa#diff-859e2b09d96c61167d13e65208236dd8R332).

Rebased on master
Include https://github.com/crystal-lang/crystal/pull/7546#discussion_r273862225 suggestion in 825b9e4d9f15151287f96e70f2d617c8558d82ec
Include https://github.com/crystal-lang/crystal/pull/7546#issuecomment-481627266 suggestion to move the call to set the stack bottom to GC.before_collect in the last commit a3632e1be4331e885bb2c7b1b8fca17d8442f903

As I mentioned in before using setting the stack bottom in the context switch or before_collect didn't show impact in the current tests. Adding the change seems reasonable.

Today we've been reviewing this PR with waj and we found a missing piece introduced in the last refactor at a14b4aaf2a56131fc31fe7bf3e3e62f2db4eea25 where for single thread we were not setting the stackbottom.
Now that is fixed at for single thread app there should be no impact.

Can we just use mt instead of preview_mt?

I prefer to use preview_mt for now. It requires less explanation about why mt is not usable yet.


Refactor formatting of leading comments

closes #53 even if the title of the ticket mentioned trailing comments.
I have tried to take blank lines between and after comment to take into account. For that I have also updated the line corrector to get enough space for the leading comments.

Unfortunately, there are other problems with the comments. I have updated the tests.

Closed in favor of #128


[DBZ-PGYB][yugabyte/yugabyte-db#23153] Change default plugin to yboutput

This PR changes the default plugin in the connector to yboutput.
This closes yugabyte/yugabyte-db#23153

What about the connector unit tests ? Have they been run ?


feat(db): add karina to list of analyst names

Implements #1285
Adds Karina to the list of Analyst Names

Name was manually added


Add Xbox Series X to repo detection

See Github issue: https://github.com/WhichBrowser/Parser-PHP/issues/627

Coverage remained the same at 100.0% when pulling c40c452d88a9d756142f0edbb8d707a79eef0db0 on ayumi-cloud:xbox-series-x into da24adc4f4f26002673d236e69b91a10f2fd594c on WhichBrowser:master.

Coverage remained the same at 100.0% when pulling c40c452d88a9d756142f0edbb8d707a79eef0db0 on ayumi-cloud:xbox-series-x into da24adc4f4f26002673d236e69b91a10f2fd594c on WhichBrowser:master.


Release 2.7

For detailed performance numbers, check #1346.

Current coverage is 100.00%

Merging #1349 into master will not affect coverage as of ad49670

@@            master   #1349   diff @@
======================================
  Files            1       1
  Stmts         3043    3043
  Branches       602     602
  Methods          0       0
======================================
  Hit           3043    3043
  Partial          0       0
  Missed           0       0

Review entire Coverage Diff as of ad49670

Powered by Codecov. Updated on successful CI builds.

looks good


Recursive types

Closing as inactionable; I see no way of pretty-printing a recursive type considering the format generated by the JSON printer.
That being said, the JSON pretty-printer does have an internal depth-limit, so encountering a recursive type shouldn't end up in a stack overflow.


Create and Update Tasks via Email

Ability to create and update tasks via email, using POP3 or IMAP. Creating tasks via would very useful to integrate KanBoard with other systems and also allow users to easily create and update tasks via email.

@oparoz I agree.... My current needs would be to only create new items but to have an email address for each project.
Doing this in Python and using the API would be quick to do but I am not sure how this would be structured in Kanboard. (I am sure what the best way would be to pole an email address without using some kind of cron-job in background. Maybe an interval based ajax call to an URL that does the processing?)

Many possibilities are available to implement that:

If you already have a STMP server like postfix, pipe everything to a script that use the Kanboard API.
That can be a cronjob that check an IMAP/POP mailbox.
That can be a third-party service like Mailgun that relay incoming emails to a Webhook: http://documentation.mailgun.com/quickstart-receiving.html

The last solution is very easy to implement but you have to use an external provider.
For creating task, the email subject can be the task title and the email body the task description...

If have tested 2) recently, works fine, until there is the first HTML Mail, that have to be transferred to markdown.

I implemented the easiest solution in the master branch: integration with third-party services. You have the choice between 3 SMTP providers: Sendgrid, Mailgun and Postmark.
More information here:

http://kanboard.net/documentation/create-tasks-by-email
http://kanboard.net/documentation/mailgun
http://kanboard.net/documentation/sendgrid
http://kanboard.net/documentation/postmark

If you want to implement the solution 1) or 2) mentioned above, you can do that with an external script that use the API or the webhooks.
At the moment, this system works only for task creation. I will see later if I need to implement notifications reply or update tasks by email.
@BlueTeck there is a library in the vendor folder to convert HTML to Markdown if you need it.

Testing task creation using Postmark integration get the following in debug log and no task created?

[2015-04-20 11:18:18] [debug] SELECT  * FROM "actions"
[2015-04-20 11:18:18] [debug] DURATION=0.0014398097991943
[2015-04-20 11:18:18] [debug] SELECT  * FROM "action_has_params"
[2015-04-20 11:18:18] [debug] DURATION=0.00012922286987305
[2015-04-20 11:18:18] [debug] SELECT COUNT(*) FROM "users"  WHERE "id" = ?
[2015-04-20 11:18:18] [debug] DURATION=0.00015807151794434
[2015-04-20 11:18:18] [debug] SQL_QUERIES=3
[2015-04-20 11:18:18] [debug] RENDERING=0.093764066696167
[2015-04-20 11:18:20] [debug] SELECT  * FROM "actions"
[2015-04-20 11:18:20] [debug] DURATION=0.0014429092407227
[2015-04-20 11:18:20] [debug] SELECT  * FROM "action_has_params"
[2015-04-20 11:18:20] [debug] DURATION=0.00012898445129395
[2015-04-20 11:18:20] [debug] SELECT  "settings"."option", "settings"."value" FROM "settings"
[2015-04-20 11:18:20] [debug] DURATION=0.00014495849609375
[2015-04-20 11:18:20] [debug] SQL_QUERIES=3
[2015-04-20 11:18:20] [debug] RENDERING=0.091457843780518

Checked From, Subject, MailboxHash and HtmlBody within Postmark -> Inbound -> Activity -> JSON and looks correct.

I just tested again with my own installation and that works for me.

Be sure to have the last version of master + composer install
Be sure the from address match the kanboard user email address
Be sure the recipient address match the project identifier<EMAIL_ADDRESS>Be sure the webhook url is correct in postmark settings
The wehbook should send back a status code 200 OK and the body "PARSED"
In the debug log file, you should see those debug messages if the message is ignored: https://github.com/fguillot/kanboard/blob/master/app/Integration/PostmarkWebhook.php#L32-L46
You can check the webhook response code in your webserver logs

This email integration looks awesome.
Would love to use it, though AFAIK this only support "public" email providers.
We, like many corporations, are not allowed to use external IT services and require to use an inhouse email system.
Therefore I'd like to request support for standard pop3/imap support.
Your thoughts on this?
Should I open a separate issue for this?
Just a note again, that this project really rocks.


First pass at a design for displaying day-of-week on list.

Can you try switching the number and day around? I feel like the day of the week is less important than the date. But maybe it looks better this way.

@kamiyo ^

Looks good to me!


Update Kafka/Kafka-connect scripts

(3) Third PR unit to review. Update scripts to what was in upstream/master
Issue: https://github.com/jboss-container-images/jboss-amqstreams-1-openshift-image/issues/1
Issue: https://github.com/jboss-container-images/jboss-amqstreams-1-openshift-image/issues/4

Changes backported, not expecting any more upstream changes for this release


Add wasm tests

I only recently realized that we're not running any tests on wasm. The normal tests (with #[test]) are not run in the wasm target. We can see that in the test output at https://github.com/PRQL/prql/actions/runs/4288400460/jobs/7470357470
Ideally we'd do at least some basic tests. I don't think we have file access in wasm, but if we included some of the examples with include_str! and checked they compiled (even without checking the result), that would be good progress. My guess is that we're going to get failures in executing functions — for example our recent issues with stack space with Chumsky — rather than a subtlety different output.

We do have tests for prql-js that run on node, so we do test the wasm module.
What exactly do you mean with "wasm tests"?

What exactly do you mean with "wasm tests"?

I had meant the tests that run on:
cargo test --target=wasm32-unknown-unknown

We do have tests for prql-js that run on node, so we do test the wasm module.

Yes, 100%, I should have included that. They're narrow, but they do cover most likely failures.
Still — adding some basic tests that run from cargo could just be a couple of test functions which iterate over our examples, and cover a much wider range of failures.
They'd also run in our rust tests, which run on every PR, so would gate any rust dependency changes.
Does that make sense to you?

We could that that with wasm-bindgen: https://rustwasm.github.io/docs/book/game-of-life/testing.html

Yes we already have that setup it seems, we just don't run any tests...
https://github.com/PRQL/prql/blob/b05c626f9b6fb00aa18e7864e6b9b44e1a06a664/.cargo/config.toml#L1-L2


Improve Benchmarking Framework

Benchmarking currently uses a statistical confidence to determine the number of iterations. However, this can lead to extremely long-running benchmarks. Hence, I suggest to limit the number of iterations by utilising benchmarking based on a max timeout.
This is the standard seen in C++ and Go. In this paradigm, fast results for fast developer iteration are more important than statistical significance.

I think we should also better separate out MSM benches in the benchmarks, the time delays are way too long for benching field arithmetic changes.

Actually, I've come to realise the long benching times comes not from the inadequacy of the benchmarking framework, but from the fact that generating the random data is done on a single thread and takes very long.
My suggestion is to use constant data which we can generate in a data file.

Alternatively, we can use a fn_once

Ah that's unforunate, but we can just parallelize the generation of the data, or maybe just sample a single input, and then mutate it somehow (eg: double it) to obtain the rest of the inputs?

doubling it isn't great, because that introduces data dependencies into the core benchmark

No I mean, instead of doing
for i in 0..1000 {
	input.push(G::rand(&mut rng)
}
bencher.iter(...);

we do
let g = G::rand(&mut rng);
for i in 0..1000 {
	input.push(*g.double_in_place());
}
bencher.iter(...);

I've used lazy static and it's insanely fast now, so I think no one will have any more complaints. I'll push changes soon

oh but for MSM we might have to use a pseudorandom distribution, generated by multiplying generator with random scalars in small range.

Actually, I realised for the particular case of MSM, bencher by default takes at least 300 samples, so it's just completely inadequate.


[Audit_RV] Rounding error for the LRNA imbalance when buying with LRNA.

The rounding error for the LRNA imbalance when buying with LRNA can be fairly large, increasing the value of LRNA (over time, as the imbalance gets paid) and reducing profits from HDX fees.
The lower bound for the LRNA imbalance rounding error can be approximated by ([x] is the unrounded value of x)
-Delta Balance_LRNA / ([hub_denominator] - 1) - something. The "something" part is usually not that large. However, in principle, the [hub_denominator] value can be rather small, e.g. it can be 2, which means that the total rounding error can be large.

fixed in #70


set config values from configmap or environment vars

This PR add a function getConfig to set configuration values from ConfigMap file or environment variable.
Working in a similar way to this python code in kepler-model-server.
def getConfig(key, default):
    # check configmap path
    file = os.path.join(CONFIG_PATH, key)
    if os.path.exists(file):
        with open(file, "r") as f:
            return f.read()
    # check env
    return os.getenv(key, default)

Signed-off-by: Sunyanan Choochotkaew<EMAIL_ADDRESS>

@marceloamaral ready to merge?

lgtm
@marceloamaral any more change needed?


segfault crash on reload

As requested on IRC:
coredumpctl gdb sway
           PID: 14513 (sway)
           UID: 1000 (milk)
           GID: 1000 (milk)
        Signal: 11 (SEGV)
     Timestamp: Mon 2019-04-29 21:36:08 BST (17min ago)
  Command Line: sway
    Executable: /usr/bin/sway
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (milk)
       Boot ID: 909207c119dd4b30ba9ef60b6ff4b803
    Machine ID: 868484078e134905a6dcbc0fdd2079bb
      Hostname: rose.local
       Storage: /var/lib/systemd/coredump/core.sway.1000.909207c119dd4b30ba9ef60b6ff4b803.14513.1556570168000000.lz4
       Message: Process 14513 (sway) of user 1000 dumped core.

                Stack trace of thread 14513:
                #0  0x00007f997c4b51c3 wl_list_insert (libwayland-server.so.0)
                #1  0x00007f997c426d51 wl_signal_add (libwlroots.so.2)
                #2  0x00007f997c427820 xserver_handle_ready (libwlroots.so.2)
                #3  0x00007f997c4b2195 n/a (libwayland-server.so.0)
                #4  0x00007f997c4b27f2 wl_event_loop_dispatch (libwayland-server.so.0)
                #5  0x00007f997c4b139c wl_display_run (libwayland-server.so.0)
                #6  0x000056260ff42a0f server_run (sway)
                #7  0x000056260ff41f26 main (sway)
                #8  0x00007f997c1dfce3 __libc_start_main (libc.so.6)
                #9  0x000056260ff33f3e _start (sway)

                Stack trace of thread 14515:
                #0  0x00007f997c1a8bac pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f997499f254 n/a (i965_dri.so)
                #2  0x00007f997499ef78 n/a (i965_dri.so)
                #3  0x00007f997c1a2a92 start_thread (libpthread.so.0)
                #4  0x00007f997c2b6cd3 __clone (libc.so.6)

GNU gdb (GDB) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/sway...done.
[New LWP 14513]
[New LWP 14515]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `sway'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f997c4b51c3 in wl_list_insert () from /usr/lib/libwayland-server.so.0
[Current thread is 1 (Thread 0x7f9975225a40 (LWP 14513))]
>>> bt full
#0  0x00007f997c4b51c3 in wl_list_insert () at /usr/lib/libwayland-server.so.0
#1  0x00007f997c426d51 in wl_signal_add (signal=0x562612169798, listener=0x562612895100) at /usr/include/wayland-server-core.h:435
#2  0x00007f997c426d51 in xwm_set_seat (xwm=0x562612894d60, seat=0x5626121694e0) at ../xwayland/selection/selection.c:309
#3  0x00007f997c427820 in xserver_handle_ready (signal_number=<optimized out>, data=0x56261219d440) at ../xwayland/xwayland.c:237
        wlr_xwayland = 0x56261219d440
        stat_val = 0
#4  0x00007f997c4b2195 in  () at /usr/lib/libwayland-server.so.0
#5  0x00007f997c4b27f2 in wl_event_loop_dispatch () at /usr/lib/libwayland-server.so.0
#6  0x00007f997c4b139c in wl_display_run () at /usr/lib/libwayland-server.so.0
#7  0x000056260ff42a0f in server_run (server=0x56260ffa9ea0 <server>) at ../sway/sway/server.c:216
#8  0x000056260ff41f26 in main (argc=1, argv=0x7ffd61d93658) at ../sway/sway/main.c:397
        verbose = 0
        debug = 0
        validate = 0
        allow_unsupported_gpu = 0
        long_options = {[0] = {
            name = 0x56260ff8ef60 "help",
            has_arg = 0,
            flag = 0x0,
            val = 104
          }, [1] = {
            name = 0x56260ff8ef65 "config",
            has_arg = 1,
            flag = 0x0,
            val = 99
          }, [2] = {
            name = 0x56260ff8ef6c "validate",
            has_arg = 0,
            flag = 0x0,
            val = 67
          }, [3] = {
            name = 0x56260ff8ef75 "debug",
            has_arg = 0,
            flag = 0x0,
            val = 100
          }, [4] = {
            name = 0x56260ff8ef7b "version",
            has_arg = 0,
            flag = 0x0,
            val = 118
          }, [5] = {
            name = 0x56260ff8ef83 "verbose",
            has_arg = 0,
            flag = 0x0,
            val = 86
          }, [6] = {
            name = 0x56260ff8ef8b "get-socketpath",
            has_arg = 0,
            flag = 0x0,
            val = 112
          }, [7] = {
            name = 0x56260ff8ef9a "unsupported-gpu",
            has_arg = 0,
            flag = 0x0,
            val = 117
          }, [8] = {
            name = 0x56260ff8efaa "my-next-gpu-wont-be-nvidia",
            has_arg = 0,
            flag = 0x0,
            val = 117
          }, [9] = {
            name = 0x0,
            has_arg = 0,
            flag = 0x0,
            val = 0
          }}
        config_path = 0x0
        usage = 0x56260ff8ea28 "Usage: sway [options] [command]\n\n  -h, --help", ' ' <repeats 13 times>, "Show help message and quit.\n  -c, --config <config>  Specify a config file.\n  -C, --validate         Check the validity of the config file, th"...
        c = -1

Archlinux package user also experiencing a similar crash https://s.natalian.org/2019-05-11/bt.txt

Can confirm same crash on a arch system.  Updated as of yesterday.  Sway version 1.4 package shows sway 1:1.4-9
$ coredumpctl debug sway
           PID: 20588 (sway)
           UID: 1000 (redacted)
           GID: 1000 (redacted)
        Signal: 11 (SEGV)
     Timestamp: Wed 2020-06-17 13:02:19 AKDT (5min ago)
  Command Line: sway
    Executable: /usr/bin/sway
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (redacted)
       Boot ID: redacted
    Machine ID: redacted
      Hostname: redacted
       Storage: /var/lib/systemd/coredump/core.sway.1000.redacted
       Message: Process 20588 (sway) of user 1000 dumped core.

                Stack trace of thread 20588:
                #0  0x00007f364fdb74b5 __strlen_avx2 (libc.so.6 + 0x1624b5)
                #1  0x00007f364fce4233 __strdup (libc.so.6 + 0x8f233)
                #2  0x0000560bec6af442 n/a (sway + 0x4f442)
                #3  0x0000560bec691b6f n/a (sway + 0x31b6f)
                #4  0x0000560bec694961 n/a (sway + 0x34961)
                #5  0x00007f364f6611dd wl_event_loop_dispatch (libwayland-server.so.0 + 0xb1dd)
                #6  0x00007f364f65f4e7 wl_display_run (libwayland-server.so.0 + 0x94e7)
                #7  0x0000560bec66f5ff n/a (sway + 0xf5ff)
                #8  0x00007f364fc7c002 __libc_start_main (libc.so.6 + 0x27002)
                #9  0x0000560bec66f94e n/a (sway + 0xf94e)

                Stack trace of thread 20590:
                #0  0x00007f364f520e32 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xfe32)
                #1  0x00007f364d9963bc n/a (iris_dri.so + 0x4ae3bc)
                #2  0x00007f364d994db8 n/a (iris_dri.so + 0x4acdb8)
                #3  0x00007f364f51a422 start_thread (libpthread.so.0 + 0x9422)
                #4  0x00007f364fd54bf3 __clone (libc.so.6 + 0xffbf3)

                Stack trace of thread 20591:
                #0  0x00007f364f520e32 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xfe32)
                #1  0x00007f364d9963bc n/a (iris_dri.so + 0x4ae3bc)
                #2  0x00007f364d994db8 n/a (iris_dri.so + 0x4acdb8)
                #3  0x00007f364f51a422 start_thread (libpthread.so.0 + 0x9422)
                #4  0x00007f364fd54bf3 __clone (libc.so.6 + 0xffbf3)

                Stack trace of thread 20592:
                #0  0x00007f364f520e32 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xfe32)
                #1  0x00007f364d9963bc n/a (iris_dri.so + 0x4ae3bc)
                #2  0x00007f364d994db8 n/a (iris_dri.so + 0x4acdb8)
                #3  0x00007f364f51a422 start_thread (libpthread.so.0 + 0x9422)
                #4  0x00007f364fd54bf3 __clone (libc.so.6 + 0xffbf3)

                Stack trace of thread 20593:
                #0  0x00007f364f520e32 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xfe32)
                #1  0x00007f364d9963bc n/a (iris_dri.so + 0x4ae3bc)
                #2  0x00007f364d994db8 n/a (iris_dri.so + 0x4acdb8)
                #3  0x00007f364f51a422 start_thread (libpthread.so.0 + 0x9422)
                #4  0x00007f364fd54bf3 __clone (libc.so.6 + 0xffbf3)

GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/sway...
(No debugging symbols found in /usr/bin/sway)
[New LWP 20588]
[New LWP 20590]
[New LWP 20591]
[New LWP 20592]
[New LWP 20593]

warning: Unexpected size of section `.reg-xstate/20588' in core file.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `sway'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section `.reg-xstate/20588' in core file.
#0  0x00007f364fdb74b5 in __strlen_avx2 () from /usr/lib/libc.so.6
[Current thread is 1 (Thread 0x7f364ec74dc0 (LWP 20588))]
(gdb) bt full
#0  0x00007f364fdb74b5 in __strlen_avx2 () at /usr/lib/libc.so.6
#1  0x00007f364fce4233 in strdup () at /usr/lib/libc.so.6
#2  0x0000560bec6af442 in  ()
#3  0x0000560bec691b6f in  ()
#4  0x0000560bec694961 in  ()
#5  0x00007f364f6611dd in wl_event_loop_dispatch () at /usr/lib/libwayland-server.so.0
#6  0x00007f364f65f4e7 in wl_display_run () at /usr/lib/libwayland-server.so.0
#7  0x0000560bec66f5ff in  ()
#8  0x00007f364fc7c002 in __libc_start_main () at /usr/lib/libc.so.6
#9  0x0000560bec66f94e in  ()
(gdb)

@KL1RL Your backtrace is different than the others. Unfortunately, it's missing debug symbols so it will be difficult to pin point the issue. Can you please compile sway from source? It's possible that it's already been fixed on sway master. If you are able to replicate, please open a sway issue with the backtrace and a debug log.


Fixing check-commits-count to work on other branches than main.

Signed-off-by: Yoni Bettan<EMAIL_ADDRESS>
/cc @qbarrand

Codecov Report
Base: 77.76% // Head: 77.90% // Increases project coverage by +0.13% :tada:

Coverage data is based on head (082a839) compared to base (5d0daa6).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #270      +/-   ##
==========================================
+ Coverage   77.76%   77.90%   +0.13%
==========================================
  Files          26       26
  Lines        2789     2792       +3
==========================================
+ Hits         2169     2175       +6
+ Misses        513      511       -2
+ Partials      107      106       -1

Impacted Files
Coverage Δ

internal/cluster/cluster.go
88.35% <0.00%> (+2.34%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

/hold
It might not fix the issue

/hold
It might not fix the issue

After researching and internal discussions it seems that this change should work as expected. To sum up, the CI checks out the target branch, merges the PR commits on top of the latter and continues with the respective test.

/unhold

/lgtm


fix new without delete

Called new in constructor
new epoll_event;

but didn't release the resource in destructor.

Perhaps it would have been better off as a unique_ptr, but this will do for now :+1:

Or can we just use epoll_event directly without pointer? There's only one event in tap driver for file descriptor tun_fd.
I was checking how IncludeOS uses epoll and surprised to find it allocate the epoll_event on the heap for one event. That's why I found the bug.

Yea, you are just looking at my pet project - running the OS on linux. I dont have much experience with epoll, but I wanted to use it to make sure the performance was at least decent/comparable to running in a VM. I also have trouble using more than one TAP device. PLEASE HELP :pray:


Support granule refcount and graunule transaction

First of all, sorry for a huge PR.. but I thought it would be better to put them in all at once.
Feature-1:  granule transaction
What it means and examples
"granule transaction" means typical stuff that requires "transaction management".
Take the example of "REALM_CREATE".
listen!(mainloop, rmi::REALM_CREATE, |arg, ret, rmm| {
    let mut granule = get_granule_if!(arg[0], GranuleState::Delegated);  // (1) get_granule
    ....  // (2) do something with granule
    set_granule(&mut granule, GranuleState::RD)   // (3) set_granule at the end of this handler when everything is ok.
}

We can say that (1)~(3) is a granule transaction to access a granule in a concurrency-safe manner.

let mut granule = get_granule_if!(addr, state):
Get a granule if the state of it equals to the given state.
set_granule(&mut granule, GranuleState::RD):
At the end of a function (in more general, a point where you want to terminate a transaction),
you can set the state of the granule.

The reason why it is needed is Granule must be accessed in a unit of transaction.
For example, we can say that the below code is problematic when some error comes out.
// current implementation
listen!(mainloop, rmi::REALM_CREATE, |arg, ret, rmm| {
    set_granule(arg[0], GranuleState::RD);  // (1) set_granule first.
    ...
       return Err(..);  // (2) error comes out, then REALM_CREATE fails but the granule is still GranuleState::RD.
                                // this is semantically wrong.
    ...
    Ok(())
}

Note for @bitboom :  mark_realm() and mark_ns() have been rewritten considering this granule transaction.
Secure API designs
Take the example of REALM_CREATE again.
listen!(mainloop, rmi::REALM_CREATE, |arg, ret, rmm| {
    let mut granule = get_granule_if!(arg[0], GranuleState::Delegated);  // (1) get_granule
     let rd = granule.content_mut::<Rd>();     // (2) get a right to access content
     rd.init(id);   // (3) write something to the physical address of RD securely.
    set_granule(&mut granule, GranuleState::RD)   // (4) set_granule at the end of this handler when everything is ok.
}

(1) let mut granule represents EntryGuard, which means this API tries to acquire a granule lock. And if successful,
it can have a guard to access a granule behind the lock. The lock is of course automatically unlocked.
Every operation can only be done through this guard. So, we can ensure that read/write from/to a granule is done under a proper lock.
Also, (2)/(3):  "content" (placed at the address of a granule) can only be read/written under the lock. As this is a way to securely access Rd/Rec/..., ManuallyDrop is no longer used for Rd/Rec.
See granule/mod.rs Line_64 for more detail about this design.

Feature-2:  refcount
At this point, the only point where refcount matters lies in REC_CREATE.
listen!(mainloop, rmi::REC_CREATE, |arg, ret, rmm| {
    let rd_granule = get_granule_if!(arg[1], GranuleState::RD);
    let mut rec_granule = get_granule_if!(arg[0], GranuleState::Delegated);
    ....
    set_granule(&mut rec_granule, GranuleState::Rec)?;
    set_granule_parent(rd_granule.clone(), &mut rec_granule)?;  // --> rd is parent and rec is child.
    Ok(())
}

If REALM_DESTROY is called when there is one or more than two RECs,
it returns Err(Error::MmIsInUse). The code to check this is in granule/entry.rs Line_144.
Note that I haven't tested calling "REALM_DESTROY" when REC is still there.
We need a system test case to do this.
TODO

Refactor "PageTable" (more precisely, move "array-based entries" from a core interface to each impl)
Improve the API design of granule transaction considering "RMM map".
Add a document page about what this PR has done.

NOTICE:  I found that realm-linux booting goes something wrong with this PR. I'll get it back to Draft and look into this issue and back to ready for review again once it gets done.


libwasmer.so: file not recognized: File format not recognized

Summary
When I build the ext, it gives me some error info:
creating libtool
appending configuration tag "CXX" to libtool
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC...
checking if g++ static flag  works... yes
checking if g++ supports -c -o file.o... yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
(cached) (cached) checking how to hardcode library paths into programs... unsupported
configure: creating ./config.status
config.status: creating config.h
/bin/sh /just/php-ext-wasm/src/libtool --mode=compile g++  -I. -I/just/php-ext-wasm/src -DPHP_ATOM_INC -I/just/php-ext-wasm/src/include -I/just/php-ext-wasm/src/main -I/just/php-ext-wasm/src -I/usr/local/php7/include/php -I/usr/local/php7/include/php/main -I/usr/local/php7/include/php/TSRM -I/usr/local/php7/include/php/Zend -I/usr/local/php7/include/php/ext -I/usr/local/php7/include/php/ext/date/lib  -DHAVE_CONFIG_H  -std=c++11   -c /just/php-ext-wasm/src/wasm.cc -o wasm.lo
mkdir .libs
 g++ -I. -I/just/php-ext-wasm/src -DPHP_ATOM_INC -I/just/php-ext-wasm/src/include -I/just/php-ext-wasm/src/main -I/just/php-ext-wasm/src -I/usr/local/php7/include/php -I/usr/local/php7/include/php/main -I/usr/local/php7/include/php/TSRM -I/usr/local/php7/include/php/Zend -I/usr/local/php7/include/php/ext -I/usr/local/php7/include/php/ext/date/lib -DHAVE_CONFIG_H -std=c++11 -c /just/php-ext-wasm/src/wasm.cc  -DPIC -o .libs/wasm.o
/bin/sh /just/php-ext-wasm/src/libtool --mode=link cc -DPHP_ATOM_INC -I/just/php-ext-wasm/src/include -I/just/php-ext-wasm/src/main -I/just/php-ext-wasm/src -I/usr/local/php7/include/php -I/usr/local/php7/include/php/main -I/usr/local/php7/include/php/TSRM -I/usr/local/php7/include/php/Zend -I/usr/local/php7/include/php/ext -I/usr/local/php7/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2    -o wasm.la -export-dynamic -avoid-version -prefer-pic -module -rpath /just/php-ext-wasm/src/modules  wasm.lo -Wl,-rpath,/just/php-ext-wasm/src/. -L/just/php-ext-wasm/src/. -lwasmer
cc -shared  .libs/wasm.o  -L/just/php-ext-wasm/src/. -lwasmer  -Wl,-rpath -Wl,/just/php-ext-wasm/src/. -Wl,-soname -Wl,wasm.so -o .libs/wasm.so
/just/php-ext-wasm/src/./libwasmer.so: file not recognized: File format not recognized
collect2: error: ld returned 1 exit status
make: *** [wasm.la] Error 1
error: Recipe `build` failed with exit code 2

And when I try just build-runtime, the info is following:

[root@2a21bde6fff5 src]# just build-runtime
    Finished release [optimized] target(s) in 0.21s
ln: failed to create symbolic link 'wasmer/libwasmer_runtime_c_api.so': No such file or directory
error: Recipe `build-runtime` failed with exit code 1

I create a wasmer dir, it works:

[root@2a21bde6fff5 php-ext-wasm]# just build-runtime
    Finished release [optimized] target(s) in 0.09s

Then run just build the error info is:

creating libtool
appending configuration tag "CXX" to libtool
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC...
checking if g++ static flag  works... yes
checking if g++ supports -c -o file.o... yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
(cached) (cached) checking how to hardcode library paths into programs... unsupported
configure: creating ./config.status
config.status: creating config.h
/bin/sh /just/php-ext-wasm/src/libtool --mode=compile g++  -I. -I/just/php-ext-wasm/src -DPHP_ATOM_INC -I/just/php-ext-wasm/src/include -I/just/php-ext-wasm/src/main -I/just/php-ext-wasm/src -I/usr/local/php7/include/php -I/usr/local/php7/include/php/main -I/usr/local/php7/include/php/TSRM -I/usr/local/php7/include/php/Zend -I/usr/local/php7/include/php/ext -I/usr/local/php7/include/php/ext/date/lib  -DHAVE_CONFIG_H  -std=c++11 -fPIC   -c /just/php-ext-wasm/src/wasm.cc -o wasm.lo
mkdir .libs
 g++ -I. -I/just/php-ext-wasm/src -DPHP_ATOM_INC -I/just/php-ext-wasm/src/include -I/just/php-ext-wasm/src/main -I/just/php-ext-wasm/src -I/usr/local/php7/include/php -I/usr/local/php7/include/php/main -I/usr/local/php7/include/php/TSRM -I/usr/local/php7/include/php/Zend -I/usr/local/php7/include/php/ext -I/usr/local/php7/include/php/ext/date/lib -DHAVE_CONFIG_H -std=c++11 -fPIC -c /just/php-ext-wasm/src/wasm.cc  -DPIC -o .libs/wasm.o
/bin/sh /just/php-ext-wasm/src/libtool --mode=link cc -DPHP_ATOM_INC -I/just/php-ext-wasm/src/include -I/just/php-ext-wasm/src/main -I/just/php-ext-wasm/src -I/usr/local/php7/include/php -I/usr/local/php7/include/php/main -I/usr/local/php7/include/php/TSRM -I/usr/local/php7/include/php/Zend -I/usr/local/php7/include/php/ext -I/usr/local/php7/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2    -o wasm.la -export-dynamic -avoid-version -prefer-pic -module -rpath /just/php-ext-wasm/src/modules  wasm.lo -Wl,-rpath,/just/php-ext-wasm/src/. -L/just/php-ext-wasm/src/. -lwasmer
cc -shared  .libs/wasm.o  -L/just/php-ext-wasm/src/. -lwasmer  -Wl,-rpath -Wl,/just/php-ext-wasm/src/. -Wl,-soname -Wl,wasm.so -o .libs/wasm.so
/just/php-ext-wasm/src/./libwasmer.so: file not recognized: File format not recognized
collect2: error: ld returned 1 exit status
make: *** [wasm.la] Error 1
error: Recipe `build` failed with exit code 2

Additional details

PHP 7.3
CentOS 7
cargo 1.42.0 (86334295e 2020-01-31)

I already know how the solve this problem, change the branch and compile:
git checkout trying
./build.sh

Look at build.sh detail.

What was the issue @suhanyujie ?

What was the issue @suhanyujie ?

It's finished. I'll just close the issue.

I've to apologize, I totally missed your issue! Sorry for that. If there is anything I can help you with, feel free to ping.

I am still not able to build it in the master branch. Using the branch "trying" helped.
#uname -a
Linux fae8cd9b7f2d 4.19.121-linuxkit #1 SMP Tue Dec 1 17:50:32 UTC 2020 x86_64 GNU/Linux
cat /etc/issue
Debian GNU/Linux 10 \n \l

I am still not able to build it in the master branch. Using the branch "trying" helped.
#uname -a
Linux fae8cd9b7f2d 4.19.121-linuxkit #1 SMP Tue Dec 1 17:50:32 UTC 2020 x86_64 GNU/Linux
cat /etc/issue
Debian GNU/Linux 10 \n \l


Add new divination cards in 3.5

#894

Any chance we could keep the list alphabetized? 👼

Good catch, #896


[KYUUBI #5867] HiveEngine support run on YARN mode

:mag: Description
Issue References 🔗

This PR aims to support hive engine run on yarn mode, close https://github.com/apache/kyuubi/issues/5867
Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Types of changes :bookmark:

[ ] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪
Behavior Without This Pull Request :coffin:
Behavior With This Pull Request :tada:
Related Unit Tests

Checklists
📝 Author Self Checklist

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[x] This patch was not authored or co-authored using Generative Tooling

📝 Committer Pre-Merge Checklist

[x] Pull request title is okay.
[x] No license issues.
[x] Milestone correctly set?
[x] Test coverage is ok
[x] Assignees are selected.
[x] Minimum number of approvals
[x] No changes are requested

Be nice. Be informative.

@pan3793 Please take a look if you find a time

Codecov Report
Attention: 290 lines in your changes are missing coverage. Please review.

Comparison is base (a0fdead) 61.28% compared to head (5a36d6e) 60.96%.
Report is 4 commits behind head on master.

Files
Patch %
Lines

...kyuubi/engine/deploy/EngineYarnModeSubmitter.scala
11.73%
198 Missing and 5 partials :warning:

...yuubi/engine/hive/HiveYarnModeProcessBuilder.scala
0.00%
59 Missing :warning:

...a/org/apache/kyuubi/engine/deploy/DeployMode.scala
10.00%
9 Missing :warning:

...apache/kyuubi/engine/hive/HiveProcessBuilder.scala
18.18%
9 Missing :warning:

...pache/kyuubi/engine/KyuubiApplicationManager.scala
0.00%
4 Missing and 1 partial :warning:

...ala/org/apache/kyuubi/util/KyuubiHadoopUtils.scala
50.00%
1 Missing and 1 partial :warning:

...in/scala/org/apache/kyuubi/config/KyuubiConf.scala
97.87%
0 Missing and 1 partial :warning:

...apache/kyuubi/service/TBinaryFrontendService.scala
0.00%
1 Missing :warning:

...ain/scala/org/apache/kyuubi/engine/EngineRef.scala
0.00%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #5868      +/-   ##
============================================
- Coverage     61.28%   60.96%   -0.32%
  Complexity       23       23
============================================
  Files           609      616       +7
  Lines         36282    36981     +699
  Branches       4976     5102     +126
============================================
+ Hits          22235    22546     +311
- Misses        11655    12014     +359
- Partials       2392     2421      +29

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks @cxzl25 , will issues mentioned in your comments will be fixed

Tested locally with https://github.com/awesome-kyuubi/hadoop-testing
overall LGTM, some minor things nice to have (it depends on you to fix it in-place or postpone the follow-up PRs)

we should support proxy user as spark engine does (should be an independent PR)
can we use the same rule to generate the engine app name
maybe engine type should be KYUUBI ?
set the tags, especially engine_ref_id, so that Kyuubi server could find it

I just found out we can leverage the YARN mode to address another existing issue of Hive engine: the Apache Hive released binary tarball ships an old version of Scala and Spark jars, we can exclude those jars to avoid potential class conflict issues.

Comparison is base (c6177ab) 61.57% compared to head (5a36d6e) 60.96%.

Can we also improve the test coverage?

Comparison is base (c6177ab) 61.57% compared to head (5a36d6e) 60.96%.

Can we also improve the test coverage?

Sure

LGTM. Left a few minor comments and hope they can be resovled in following PRs.

@Yikf Thanks, merged to master. we can address the minor comments in follow-up PRs.


Fix mouse/touch interaction anchors when using css transform scale()

closes #14179
closes #13283

@mike-000
I found some issues with my first approach and rewrote the PR to use View.getViewportSize_(), but since it is a private function of View i moved the helper function into View.getViewportScale().
I have tested this on the example implementations and also on a custom implementation in another project and it works fine for me.
Not sure I have covered ALL interactions possibly affected though.

Thanks for your effort on this, @didkan. I'm not sure this is the right approach. Any transform (e.g. also a rotation, or - more generally - a transform matrix on a dom element) can cause this kind of distortion, not just a scale.
I think this could be solved by using offsetX and offsetY instead of clientX and clientY. Such a change would be a major rework, but would simplify the code base in some places, because we wouldn't have to deal with getBoundingClientRect().


Startup Issue

Any version past 1.0.3 will not open up and just cause an error

Does this happen when opening the editor itself or when you open a project?

The editor itself after installing ethier version 1.0.4 and 1.0.5
I had also tried in both cases to uninstall and remove the folder to install it fresh and it still didn't work

fixed in v1.0.6


Data uri

Would it be possible to pass through a data uri string instead of a file?
I'm working on a project right now and have a form that uses js to attach an image to it before being submitted. The js component converts the image to base64 and attaches the string to the form.

@dopyoman Sorry its not possible. PRs are welcome.


Fixes issue in Device access token request

The fix makes the Device access token request aligned with the specification : rfc8628
Closes #1751
Summary
This fixes a bug in Device access token request, where the request always leads to the following error : unexpected error in device flow: invalid_client . This happens because of the missing client_id in the request. client_id is a required parameter as per the specification
Also the scope is incorrectly set to a slice of string :"openid", "email"  instead of one single string "openid email". This led to following error : unexpected error in device flow: invalid_request
This can be tested using --fulcio-auth-flow=device and with an OIDC provider ( that supports device code grant , example Keycloak) configured.
Release Note
NONE

Hey @rishabhsvats, sorry, cleaning up the open PRs. This LGTM, can you rebase to kick off tests again?

Hello @haydentherapper ,  I have rebased, could you approve the workflow approval ?

ptal @haydentherapper


Close Tag addon doesn't work properly

Hi,
I found this bug in another project: hackmdio/hackmd#400.
Developer of HackMD said this bug may cause by CodeMirror's Close Tag addon,
this link is a demo page in HackMD for reproduce the bug.
About the bug

If I type someword I want to delete:
I want to have delete line on this sentence.

And I put <s> in the beginning like following:
<s>I want to have delete line on this sentence.

Here will automatically show the close tag with inproperly content like following:
<s></sI>I want to have delete line on this sentence.

I thought the word I shouldn't appear in closetag </s>

Thanks a lot 😄

This doesn't appear to be happening on http://codemirror.net/demo/closetag.html , so I guess HackMD is running an old or modified version of the code.


How to serialize rule configuration to json config

Hello,
We are using FluentValidation in our ASP.NET MVC Web API project to perform server validation.
We want to use FluentValidation rule configuration as the single source of truth for validation logic which will be shared with client. To pass rule configuration to the client we have to serialize rules configuration for the concert type to json.
Do we have opportunity to serialize FluentValidation rule config to json?
Regards,
Siarhei

Hi, no this isn't possible I'm afraid.
On Thursday, 28 July 2016, Siarhei Zhalezka<EMAIL_ADDRESS>wrote:

Hello,
We are using FluentValidation in our ASP.NET MVC Web API project to
perform server validation.
We want to use FluentValidation rule configuration as the single source of
truth for validation logic which will be shared with client. To pass rule
configuration to the client we have to serialize rules configuration for
the concert type to json.
Do we have opportunity to serialize FluentValidation rule config to json?
Regards,
Siarhei
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/JeremySkinner/FluentValidation/issues/305, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAFgEo1kj8AGXfT4lvgkk13zQx1QdSk3ks5qaGHVgaJpZM4JW9Us
.


Create simple.ps1

Simple method of using Windows version of Selenium. Might need to look at switching over, as had a lot of issues using the built-in code.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[WIP][SPARK-32708] Query optimization fails to reuse exchange with DataSourceV2

What changes were proposed in this pull request?
Override doCanonicalize function of class DataSourceV2ScanExec

Why are the changes needed?

Query plan of DataSourceV2 fails to reuse any exchange. This change can make DataSourceV2's plan more optimized and reuse exchange as DataSourceV1 and parquet do.
Direct reason: equals function of DataSourceV2ScanExec returns 'false' as comparing the same V2 scans(same table, outputs and pushedfilters)
Actual cause : With query plan's default doCanonicalize function, pushedfilters of DataSourceV2ScanExec are not canonicalized correctly. Essentially expressionId of predicate columns are not normalized.
Spark 32708 was not caught by my tests previously added for [SPARK-32609] because the above issue happens only when the same filtered column are of different expression id (eg :  join table t1 with t1)
Does this PR introduce any user-facing change?

no
How was this patch tested?

unit test added

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

2.4?

Test build #127968 has started for PR 29564 at commit 704efbc.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32594/
Test PASSed.

If the issue does not happen in branch-3.0+, I think we might need to check which commit's resolved it from the commit history first. If it found, we might be able to just cherry-pick it.

Test build #127968 has finished for PR 29564 at commit 704efbc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127968/
Test PASSed.

In Branch 3.0, there is a mixed-in trait SupportsPushDownFilters which is introduced by https://github.com/apache/spark/pull/19136 and https://github.com/apache/spark/pull/19424 .
However, if we are going to cherry-pick the PRs mentioned above, then there will be reasons to cherry-pick the other data source V2 related PRs into 2.4 as well.
@mingjialiu is there a strong reason to use data source v2 on branch 2.4 instead of 3.0? There seems quite some work to sync DS v2 api changes to branch 2.4
cc @cloud-fan as well.

If exchange reuse is broken, it means plan equality is broken somewhere. I think Seq[Expression] is OK as long as we canonicalize it before comparing it. FileSourceScanExec also contains Seq[Expression] and it's fine.
Can you look into it more and have a more surgical fix?

Test build #128481 has started for PR 29564 at commit a6e4709.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33105/
Test PASSed.

If exchange reuse is broken, it means plan equality is broken somewhere. I think Seq[Expression] is OK as long as we canonicalize it before comparing it. FileSourceScanExec also contains Seq[Expression] and it's fine.
Can you look into it more and have a more surgical fix?

Updated with a more surgical fix. Please review.

In Branch 3.0, there is a mixed-in trait SupportsPushDownFilters which is introduced by #19136 and #19424 .
However, if we are going to cherry-pick the PRs mentioned above, then there will be reasons to cherry-pick the other data source V2 related PRs into 2.4 as well.
@mingjialiu is there a strong reason to use data source v2 on branch 2.4 instead of 3.0? There seems quite some work to sync DS v2 api changes to branch 2.4
cc @cloud-fan as well.

My org still relies heavily on 2.4

@mingjialiu the new fix looks more reasonable. Could you add test case for the changes?

The fix LGTM, can you add a test?

Test build #128481 has finished for PR 29564 at commit a6e4709.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128481/
Test FAILed.

The fix LGTM, can you add a test?

Hi, it's a bit tricky to repro in unit test. Can I get some pointers on populating different expression ids for the same column?

CAN'T repro example in unit test:
val df = spark.read.format(classOf[AdvancedDataSourceV2].getName).load()
val q1 = df.select(($"i" + 1).as("k"), ($"i" - 1).as("j")).filter('i > 5)
val q2 = df.select(($"i" + 1).as("k"), ($"i" - 1).as("j")).filter('i > 5)
val scans1 = getV2ScanExecs(q1.join(q2, "j"))
assert(scans1(0).sameResult(scans1(1)))
scans1(0).sameResult(scans1(1)) will always return true even if filtered columns are not properly canonicalized (as circled in screenshots)

Hi,
Regarding test coverage,  it's a bit tricky to repro in a unit test.
Can I get some pointers on populating different expression ids for the same
column? Or test suggestions?

CAN'T repro example in unit test:

val df = spark.read.format(classOf[AdvancedDataSourceV2].getName).load()
val q1 = df.select(($"i" + 1).as("k"), ($"i" - 1).as("j")).filter('i >

val q2 = df.select(($"i" + 1).as("k"), ($"i" - 1).as("j")).filter('i >

val scans1 = getV2ScanExecs(q1.join(q2, "j"))
assert(scans1(0).sameResult(scans1(1)))
scans1(0).sameResult(scans1(1)) will always return true even if filtered
columns are not properly canonicalized (as circled in screenshots)

[image: other_canonicalized.png]
[image: this_canonicalized.png]

CAN repro query :  SPARK-32708
https://issues.apache.org/jira/browse/SPARK-32708

The key to repro is to have d_day_name and d_year  assigned different
expression Ids.
Relative implementation : preserve old expressionId if column not found
https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala#L285
Thank you,
Mingjia
On Wed, Sep 9, 2020 at 11:05 PM Wenchen Fan<EMAIL_ADDRESS>wrote:

The fix LGTM, can you add a test?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/29564#issuecomment-690008778, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AEMVJMYZD527O7EY27RUISLSFBUCZANCNFSM4QNQGG5Q
.

Test build #128541 has started for PR 29564 at commit 98483c8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33166/
Test PASSed.

The fix LGTM, can you add a test?

Regarding test coverage,  it's a bit tricky to repro in a unit test. Can I get some pointers on populating different expression ids for the same column? Or test suggestions?
The key to repro is to have the same column assigned different expression Ids.
Relative implementation : preserve old expressionId if column not found
Explained details in email.

The fix LGTM, can you add a test?

Test added. Please review.

The fix LGTM, can you add a test?

Regarding test coverage,  it's a bit tricky to repro in a unit test. Can I get some pointers on populating different expression ids for the same column? Or test suggestions?
The key to repro is to have the same column assigned different expression Ids.
Relative implementation : preserve old expressionId if column not found
Explained details in email.

Please ignore this message. I figured out that column's expression id is consistent within the same df.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33168/
Test PASSed.

Test build #128543 has started for PR 29564 at commit 8b864e7.

Test build #128541 has finished for PR 29564 at commit 98483c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128541/
Test PASSed.

Test build #128543 has finished for PR 29564 at commit 8b864e7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128543/
Test PASSed.

Test build #128574 has started for PR 29564 at commit acadafe.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33199/
Test PASSed.

test pyspark.mllib.tests.StreamingLogisticRegressionWithSGDTests.test_training_and_prediction is failing. @gatorsmile @cloud-fan @gengliangwang  do you think the failure is related to this change?  If yes, any suggestions on how to fix it?

I think we can rely on the jenkins test result as well

Test build #128574 has finished for PR 29564 at commit acadafe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128574/
Test PASSed.

@cloud-fan @gatorsmile  Hi, it looks to me that 1 approval is not enough for merging. Can you please approve this PR if everything looks good to you?

Thanks, merging to branch 2.4

Thank you, @mingjialiu and all!

@gengliangwang . Could you update Fix Version of SPARK-32708?

@dongjoon-hyun sure.

@gengliangwang . The Fix Version seems to be empty still.

@dongjoon-hyun Sorry, it's updated now.

Thank you, @gengliangwang !


Package request: netron

Project description
Netron is a viewer/visualizer for neural network, deep learning and machine learning models. It supports ONNX, TensorFlow Lite, Core ML, Keras, Caffe, Darknet, PyTorch, TensorFlow.js, Safetensors and NumPy with experimental support for TorchScript, TensorFlow, MXNet, OpenVINO, RKNN, ML.NET, ncnn, MNN, PaddlePaddle, GGUF and scikit-learn.
Metadata

homepage URL: https://netron.app
source URL: https://github.com/lutzroeder/netron
license: mit
platforms: linux, darwin

any updates?


[BUG] More than 2 emojis in the title editor bring weird characters

Project Information

Repository: https://github.com/esteemapp/esteem-surfer
Project Name: eSteem Surfer 2 – Steem Desktop Client

Expected behavior
My expectation was that I could add several smileys / emojis to the title of my post without any problems and without strange special characters.
Actual behavior
If you enter several smileys / emojis in the title of your post one after the other, strange characters appear from the second emoji. This only happens in the title of the post. In the post itself it works great. The video shows it quite well.
How to reproduce
Download the latest version of the Desktop Surfer app from Github. Install the software. Launch the app. Enter a pin. Switch to the editor and enter smileys several times in the title. For example the Muscle Emoji from the Screen Record.

Browser/App version: Latest App Version 2.0.5
Operating system: Mac OSX 10.14

This Issue is also on the Windows Application. Just downloaded a fresh Copy to my Notebook to check that out. Also the Problem. Mobile Version works fine.
Recording Of The Bug

GitHub Account
https://github.com/louis88-steemit

nice finding @louis88-steemit We'll fix it soon.


Fix exr_threads value -1 to disable thread pools not working.

29766348b27a2030820147561ef47cf44835c595 never actually worked, because the -1 value was getting clamped.

Makes sense, LGTM.

Backported to RB-1.7


Add option to send on transaction commit

Unsure of the best practices with regards to documentation, so please do say if I could improve this.

Is this ready to merge? I think I've addressed all the comments.

Enjoy :)

@lamby thanks! Can we have a release on PyPI please 😄

Sure. 2.2.0...

@lamby sorry to keep bothering, but it looks like 2.2.0 doesn't include these changes? Could you investigate?

@danpalmer Argh, my bad- I didn't pull locally first as I merged on the Github interface itself :)  Sorry about that...

(fixed)

Thanks!


Color

https://stackoverflow.com/questions/287871/how-do-i-print-colored-text-to-the-terminal

It is already possible to use colors, as specified in the Usage section of the project's README.

It is already possible to use colors, as specified in the Usage section of the project's README.


a note on maven target folder

What are the meaning of different folder names in the target folder?
$ ls target

antrun   dependency-maven-plugin-markers  hadoop-test  maven-archiver                  maven-status                 SystemDS.jar
classes  generated-sources                lib          maven-shared-archive-resources  systemds-2.2.0-SNAPSHOT.jar  test-classes

lib - all the dependencies in jar format


Generating  instance segmentation .json file

I don't understand how to use mask rcnn on the COCO dataset to generate the coarse segmentation results. I don't see it described this way on the Mask R-CNN repo. What specific step do I take to create the 'mask_rcnn_r50.train.segm.json' dataset?  Any clarification would be greatly appreciated!

HI, we generated those json files using mmdetection. You can refer to the 6th example in https://mmdetection.readthedocs.io/en/stable/1_exist_data_model.html#examples.

Thank you!

I see that the example they give produces the segmented mask results for COCO val set, how do we specify to test on the train set to produce the mask_rcnn_r50.train.segm.json?

Hi, for question 1, you can do this by slightly modifying the configure file of mmdet, for example:
data = dict(
    imgs_per_gpu=1,
    workers_per_gpu=2,
    test=dict(
        type=dataset_type,
        ann_file=data_root +
        # 'annotations/instancesonly_filtered_gtFine_test.json',      # inference on test set
        # img_prefix=data_root + 'leftImg8bit/test/',
        # 'annotations/instancesonly_filtered_gtFine_val.json',       # inference on val set
        # img_prefix=data_root + 'leftImg8bit/val/',
        'annotations/instancesonly_filtered_gtFine_train.json',       # inference on train set
        img_prefix=data_root + 'leftImg8bit/train/',
        pipeline=test_pipeline))

For question 2, you can generate a dummy annotation file that does not contain the 'annotations' field. Our script needs this file for information such as image file list and sizes. You can refer to 'instancesonly_filtered_gtFine_test.json' in https://github.com/open-mmlab/mmdetection/blob/master/tools/dataset_converters/cityscapes.py as an example.

Dear author, hello: The link you provided cannot be opened. Could you please provide a reference link that can be opened？


Accessing Entity Defined in Player Function from App.jsx File

Description:
I am having an issue accessing an entity that I've defined within a Player function component from my App.jsx file. The Entity Component System (ECS) I am using is integrated with React and here's the code for my Player function component:
`export default function Player(props) {
const ECS = props.ECS
const entity1 = ECS.world.add({name: "test"})
return (
    <ECS.Entity entity={entity1}>
      <ECS.Component name="position" data={{x:1, y:1, z:0}} />
      <ECS.Component name="color" data="red" />
      <ECS.Component name="poisoned" data={true} />
      <ECS.Component name="velocity" data={{x:1, y:1, z:0}} />
      <ECS.Component name="three"/>
        <mesh ref={ref}>
          <cylinderGeometry args={[1, 1, 1]} position={{x:1,y:1,z:1}} />
          <meshStandardMaterial
            wireframe={props.wireframe}
            color={ "hotpink" }
          />
        </mesh>
      <ECS.Component name="content" data="hello world" />
        {console.log("hello world")}
    </ECS.Entity>
  );
}

`
And here's how I am trying to use it in my App.jsx file:
`function App() {
const world = new World()
const ECS = createReactAPI(world)
useEffect(()=>{
console.log(world)
}, [world])
return (
<>

<pointLight position={[10, 10, 10]} />

</ section>
</>
);
}
`
I want to gain access to the entity defined in the Player component from the App component. I've attempted to use the ECS.useCurrentEntity method, but it returned null. I've also tried using useRef, but to no avail. I am seeking a solution or guidance on the best approach to resolve this issue. Thanks in advance!

May I kindly redirect you to the Discussions tab and ask you to repost your question there, ideally with better formatting? It's hard to see if the code you pasted is missing bits, or if GitHub is omitting some of your JSX, when it's not in a code block tag. Thanks!


loader-function

Hi,
I've noticed, that we can use a loader-function instead of a component within a router-config-object. The question is, how to use it, to load a component dynamically?
Is that even supported in 1.x or do I have to use community-extensions?
Wishes,
Manfred

Lazy loading of components is not built in to 1.x, but it is supported through community-extensions such as http://oclazyload.readme.io.
I might add an example of lazy loading to the example app that uses this functionality.

Thanks for this info. And Btw: thanks for this wonderful Demo-Repository. It really helped me with this topic!!

:+1:


kodiPlugins.pvr-hts: init at 2.1.18

WIP: Add Tvheadend HTSP PVR client addon for Kodi.
cc @edwtjo @domenkozar (as kodi maintainers):  this is working but I'm using that symlink trick as it searches for the addon .so in the lib dir from the original non-wrapped kodi. Any suggestion how to workaround this more cleanly?

I'll get back to you later in the evening on this.

I think it's fine. If you want to wrap lib the same way as share then this is also ok, but then you of course need to test the other plugins after you've rebased on master.

The problem is that I already tried, but kodi tries to load the plugin .so file from the unwrapped kodi path (see relevant strace fragment from before creating the symlink):
[pid  4859] stat("/nix/store/050bds9qi4y1qjfv6qp5i7yks242j8lg-kodi-15.2/share/kodi/addons/pvr.hts/pvr.hts.so", 0x7f1355279770) = -1 ENOENT (No such file or directory)
[pid  4859] stat("/nix/store/85hzfwiynzcavy3awkay0bg2cl3g25dn-kodi-15.2/lib/kodi/addons/pvr.hts/pvr.hts.so", 0x7f1355279770) = -1 ENOENT (No such file or directory)

So I see little gain in wrapping lib for now, unless we find a way to fix this. I was wondering where did you find about KODI_HOME, I can't seem to find any description and whether not using it for 'lib' is intentional or not.

So I'm guessing you've already tried setting up LD_LIBRARY_PATH=${kodi}/lib... for the wrapper?
If I remember correctly; I guessed it when I was playing around with a fidgety gamepad, SOMETHING_HOME is such a common idiom.

I thought of using LD_LIBRARY_PATH, but kodi searches for .so files in specific paths before even trying to open them (notice the stat call in the strace above).

Then I suggest you resolve the conflicts against master and merge this.

Heh, I wanted to wait on the channel update to test before committing and now kodi segfaults (from unstable channel, no plugins enabled):
[page@leaf:~]$ kodi
/nix/store/a1svf04r51khz5sx99y81ixw5cyw244d-kodi-15.2/bin/.kodi-wrapped: line 165:  7440 Segmentation fault      "$LIBDIR/${bin_name}/${bin_name}.bin" $SAVED_ARGS

@cpages I can run kodi on 16.03pre73012.eeb2935

Without the plugin, I guess? What xorg driver are you using? Another thing to check might be to revert 3e96b76 or pass the older llvm to mesa_drivers.

I had not updated nixos, doh. Sorry for the noise, and thanks both for the feedback. Wouldn't it be nice to have kodi updated in the 15.09 branch, btw?


Slug is being generated incorrectly using workshop location

When accepting a workshop request, AMY is auto-generating a slug using the workshop start date followed by the workshop location.  For example:

This results in invalid characters appearing in the slug.  It also does not follow our standard practice which is to format the slug as YYYY-MM-DD-hostname, where the hostname is usually a shorthand version of the actual host name.  For example, a workshop on 2020-12-01 at  The University of Pennsylvania Graduate School of Education may have the slug 2020-12-01-upenn-gse.
@sheraaronhurt  @Talishask  Does it make sense for AMY to try to generate a slug in some way, or would it be better to leave that to the human that is accepting the workshop?

@maneesha definitely think this should be left for the human to decide what the slug is

@sheraaronhurt I thought it did a good job except for the end where it pulled location. If we adjust that part to pull the host domain (minus the .edu or .org part) would that work for CO workshops?

@Talishask  if it can do what you suggested that would be great! I had thought of that but thought it might be too complex to do.... but it never hurts to share our wishlist!

Cool. @maneesha Could the slug be generated by appending the prefered date to the host domain (minus the .edu or .org part)?

I think that the suggestion from @Talishask  may not work in all situations.  For example, you may have domains like:

www.example.edu
www.example.edu/library
library.example.edu
and so on

We also know sometimes people run concurrent workshops, using slugs like 2020-01-01-example-python and 2020-01-01-example-r.
I think this is probably why we'd want to keep this human generated.

I think that the suggestion from @Talishask  may not work in all situations.  For example, you may have domains like:

www.example.edu
www.example.edu/library
library.example.edu
and so on

We also know sometimes people run concurrent workshops, using slugs like 2020-01-01-example-python and 2020-01-01-example-r.
I think this is probably why we'd want to keep this human generated.

We can change the algorithm to use host name's but in a computerized version. For example:

Australian Centre for Plant Functional Genomics -> australian-centre-for-plant-functional-genomics
AAAS -> aaas
AMOS Conference (Melbourne) -> amos-conference-melbourne

The date part of the slug would stay the same, only the ending would be changed.

Thanks @pbanaszkiewicz I would like for this to be something that a human decides rather than automated.

@pbanaszkiewicz  in creating a new event from a request, the slug should remain blank, for the human to fill in.

Thanks so much @maneesha !

@maneesha Right now the help text tells the users to input YYYY-MM-DD-location. Should anything change about the text?

@lauryndbrown  That help text is good. Thanks for checking!


long delay before re_main() ends at quit

I have not been able to figure out, why it takes more than 50 seconds before re_main() ends at quit in my baresip Android app when there is accounts or no network connection.
Starting of baresip follows closely the example in https://github.com/alfredh/baresip/wiki/Using-baresip-as-a-library.  Calling of re_main() goes like this:
LOGI("running main loop ...\n");
    err = re_main(signal_handler);
    out:
    if (err) {
        LOGE("stopping UAs due to error: (%d)\n", err);
        ua_stop_all(true);
    } else {
        LOGI("main loop exit\n");
    }

and signal_handler calls re_cancel():
static void ua_exit_handler(void *arg)
{
	(void)arg;
        LOGD("ua exited -- stopping main runloop\n");
	re_cancel();
}

When the app calls ua_stop_all(false), I get to log:
11-24 09:01:59.329  9915  9949 I Baresip : running main loop ...
...
11-24 09:02:01.617  9915  9915 D Baresip : ua event (EXIT)
11-24 09:02:01.617  9915  9915 D Baresip : ua exited -- stopping main runloop
11-24 09:02:01.618  9915  9950 D Baresip : ua: stop all (forced=0)
********* here is the delay
11-24 09:02:58.622  9915  9949 I Baresip : main loop exit
11-24 09:02:58.622  9915  9949 D Baresip : closing ...
11-24 09:02:58.622  9915  9949 D Baresip : unloading modules ...

There was no accounts configured in the above test.
Any ideas why re_cancel() takes so long to have effect on re_main()?

when the SIP stack is shutting down, it is waiting for all client transactions to complete,
before calling the exit handler.
if the UA is registered, closing down reg.c/sipreg.c will send a REGISTER with
expires=0 to un-register from the SIP server (and create a SIP Client Transaction).
since you have no network, the SIP stack will never get a response and instead
timeout.
It is the same with the SIP Session module; if there is an ongoing SIP call and
the application is shutting down, sipsess will try to send SIP BYE to the peer
to be polite. But if the sending fails, it will not handle the error.
You can try to set forced=1 when calling ua_stop_all, this will still try to send
un-register but not wait for any responses, and the SIP stack should exit
immediately.

I looked a bit closer at your logs, and it looks like the "ua event EXIT" is coming from
another thread (different thread from re_main).
in this case you need to "wake up" the re_main loop thread, so that it is not
blocking. This can for example be done by sending a dummy message
using struct mqueue. When the re_main thread receives this message,
it will wake up and handle all pending events, and hopefully exit after that.
The re_main loop has only 2 types of events; FD event on a socket/pipe etc
and timers. For each event that it handles, it will update its internal state,
check the "running" flag etc.
So re_cancel must be called from the same thread that runs re_main
otherwise it will not update the states properly.

Alfred E. Heggestad writes:

I looked a bit closer at your logs, and it looks like the "ua event
EXIT" is coming from another thread (different thread from re_main).

Yes. this must have something to do with threads, since as I mentioned
earlier. the delay is there event when there are no UAs.
Now I tried like this:
JNIEXPORT void JNICALL
Java_com_tutpro_baresip_BaresipService_baresipStart(...) {
...
re_thread_enter();
err = re_main(signal_handler);
re_thread_leave();
...
}
JNIEXPORT void JNICALL
Java_com_tutpro_baresip_BaresipService_baresipStop(...) {
re_thread_enter();
ua_stop_all(force);
re_thread_leave();
return;
}
but it did not help.  So I guess I needs to try the mqueue way as you
suggested.
Should I allocate an mqueue in baresipStart function above and then send
a message to it in baresipStop() function?

in your multi-threaded app, you need to "wake up" the re_main thread somehow,
either by using struct tmr or socket or struct mqueue.
this PR is related to your issue:
https://github.com/alfredh/baresip/pull/581
The mqueue can be global. I think it should be enough to make sure that
ua_stop_all is called from re main thread (via mqueue).
if you look at your log above, you will see two columns with numbers.
this is probably the Process-ID and Thread-ID. For all the log lines that you see,
the ID should be the same for all entries. Find the first logline with a different ID,
find the code that prints this line and then move that code to mqueue.

another work-around is to start a periodic timer that fires every 1000ms :)

Alfred E. Heggestad writes:

another work-around is to start a periodic timer that fires every
1000ms :)

This is what I have currently done, but using mqueue would be a more
proper solution.  Perhaps in the future there could be a built-in global
mqueue listener.


Maintenance/update precommit

Update pre-commit packages, certifi in all services and all deviceregistry packages

"Some checks were not successful" but I'm going to merge this anyway, because this is not in use anywhere and all others are on vacation. Let's fix these failing checks when they are back.


python311Packages.odp-amsterdam: 5.3.1 -> 6.0.0

Diff: https://github.com/klaasnicolaas/python-odp-amsterdam/compare/refs/tags/v5.3.1...v6.0.0
Changelog: https://github.com/klaasnicolaas/python-odp-amsterdam/releases/tag/v6.0.0
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Priorities
Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 269740 run on x86_64-linux 1

3 packages built:

home-assistant-component-tests.garages_amsterdam
python311Packages.odp-amsterdam
python311Packages.odp-amsterdam.dist

Result of nixpkgs-review pr 269740 run on x86_64-linux 1

3 packages built:

home-assistant-component-tests.garages_amsterdam
python311Packages.odp-amsterdam
python311Packages.odp-amsterdam.dist


Should reflexivity look through foralls?

I needed to replace
apply reflexivity

with
apply (anyTD reflexivity)

in the Hanoi example, because of the error message "reflexivity may only be applied to equivalence lemmas".
Goal:
  ∀ △ l. l ≡ l
*** Exception: server failure: Object (fromList [("exception",String "no scope to end.")]) : no scope to end.

Should reflexivity look through foralls?

Logically speaking, no.  You should have to skolemize that term before applying reflexivity.  If we want to add a command that combines these steps I'm all for it, but it shouldn't be called reflexivity in my opinion.
On a related note, does anyTD reflexivity actually do what we think it does here?
Is it reducing the term to forall l. l and HERMIT is recognizing that as a complete proof, or is anyTD promoting reflexivity's "success" to the top level?
In other words, would anyTD reflexivity also work as a proof for something like forall l z. l = l && l = z?

@andygill You should be able to just end-proof (or whatever the new shell equivalent is) here. More useful than calling reflexivity directly is simplify-proof and smash.
@ecaustin reflexivity rewrites an equivalence of the form e === e to true, so anyTD reflexivity in this case should give forall t l. true... at which point you can simplify away the foralls. The only case HERMIT considers a complete proof is when the lemma is reduced to true... though the end-proof command runs simplify-proof for you... wish just iterates reflexivity and elimination rules, so you don't usually have to go that far.

Many of the proof commands still look through foralls, but I think as a design principle we should move away from this to work at a single intended level (like reflexivity does) and then promote/traverse/target with kure strategies. I'm actually working on doing this cleanup now... though I basically get to look at HERMIT on Saturdays, so it might be another week or two.

I should have the hermit-shell working, with travis builds, and the tests working, before the end of this weekend. This should help a bit.

Closing. reflexivity is a primitive that does not look through foralls.


Feat(eos_cli_config_gen): add support to shutdown individual VRF neighbors in bgp

Change Summary
At the moment it is not possible to specify shutdown on a BGP peer when peer is located inside a VRF
Related Issue(s)
Fixes #1475
Component(s) name
arista.avd.eos_cli_config_gen
Proposed changes

router_bgp:
  vrfs:
    foo:
      neighbors:
        2001:db8::1:
          shutdown: true

How to test

molecule
Checklist
User Checklist

N/A

Repository Checklist

[x] My code has been rebased from devel before I start
[x] I have read the CONTRIBUTING document.
[x] My change requires a change to the documentation and documentation have been updated accordingly.
[x] I have updated molecule CI testing accordingly. (check the box if not applicable)

This knob is fundemental for the bgp peer, so we should add this setting to the documentation table.

It's a valid point. Problem is that the global VRF context is well documented and can be extended by the shutdown column. The VRF BGP neighbor section is somehow lacking behind and only shows VRF name, RD and redistribution settings.
How to proceed @ClausHolbechArista? Should I add shutdown to Global VRF and individual VRF table?

It's a valid point. Problem is that the global VRF context is well documented and can be extended by the shutdown column. The VRF BGP neighbor section is somehow lacking behind and only shows VRF name, RD and redistribution settings.
How to proceed @ClausHolbechArista? Should I add shutdown to Global VRF and individual VRF table?

I am not sure if we changed this recently, but the current template has a single table for both global and vrf neighbors.
This is the table I am talking about:
https://github.com/aristanetworks/ansible-avd/blob/devel/ansible_collections/arista/avd/molecule/eos_cli_config_gen/documentation/devices/router-bgp-vrf-lite.md#bgp-neighbors
So my ask is just to add a shutdown column here and logic for both types of neighbors. (global and vrf)

Hi @ClausHolbechArista - I hope I met your desired expectations!

Hi @ClausHolbechArista - I hope I met your desired expectations!

@c-po Not fully, but I can fix the rest before we approve the PR.

I would move shutdown to right after VRF column
We need the field to be filled for the non-vrf neighbors too.
We need to rerun molecule afterwards.

Please run, make refresh-facts from ansible_collections/arista/avd/molecule directory. This will update all facts in other scenarios.