Storm 1.0.3 Support - MesosSupervisor Committing Suicide

Hi, All. Tremendous work in getting Storm 1.x support with this framework. So far so good with 1.0.2; thank you everyone for your contributions.
I've attempted to drop in Storm 1.0.3 and ran into an issue which I think is ultimately related to a refactor of converting some clojure code to java in the storm project, specifically the Supervisor (https://issues.apache.org/jira/browse/STORM-2018)
What's happening is that by upgrading to Storm 1.0.3 the "assigned" method is no longer being called, as a result, the MesosSupervisor is not updating its internal view of '_supervisorViewOfAssignedPorts' and is committing suicide:

https://github.com/mesos/storm/blob/9911151a0efb78203f14456b7873f94fbe5b64b5/storm/src/main/storm/mesos/MesosSupervisor.java#L261

I patched this locally, by calling the 'assigned' method from the 'confirmAssigned' method which gets called at the same frequency as 1.0.2 was calling assigned. This fixes the issue and I can put in a PR for this, but I'm wondering if folks have some context as to why assigned is no longer being called and that if this change could have unintended consequences.
Here's some stacks of the calls in 1.0.2 vs 1.0.3 (I just threw exceptions in the code, and printed the trace):
1.0.2:
2017-07-21 21:51:15.616 s.m.MesosSupervisor [INFO] confirmAssigned, stack = java.lang.Throwable
	at storm.mesos.MesosSupervisor.confirmAssigned(MesosSupervisor.java:114)
	at org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9163$fn__9167.invoke(supervisor.clj:548)
	at org.apache.storm.util$filter_key$fn__469.invoke(util.clj:318)
	at clojure.core$filter$fn__4580.invoke(core.clj:2690)
	at clojure.lang.LazySeq.sval(LazySeq.java:40)
	at clojure.lang.LazySeq.seq(LazySeq.java:49)
	at clojure.lang.RT.seq(RT.java:507)
	at clojure.core$seq__4128.invoke(core.clj:137)
	at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
	at clojure.core.protocols$fn__6506.invoke(protocols.clj:101)
	at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13)
	at clojure.core$reduce.invoke(core.clj:6519)
	at clojure.core$into.invoke(core.clj:6600)
	at org.apache.storm.util$filter_key.invoke(util.clj:318)
	at org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9163.invoke(supervisor.clj:548)
	at org.apache.storm.event$event_manager$fn__8735.invoke(event.clj:40)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.lang.Thread.run(Thread.java:745)

2017-07-21 21:51:15.617 s.m.MesosSupervisor [INFO] assigned = [31001]
2017-07-21 21:51:15.618 s.m.MesosSupervisor [INFO] assigned, stack = java.lang.Throwable
	at storm.mesos.MesosSupervisor.assigned(MesosSupervisor.java:72)
	at org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9163.invoke(supervisor.clj:592)
	at org.apache.storm.event$event_manager$fn__8735.invoke(event.clj:40)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.lang.Thread.run(Thread.java:745)

1.0.3 (with my patch where confirmAssigned calls assigned):
2017-07-21 22:06:12.003 o.a.s.d.s.Supervisor [INFO] Starting supervisor with id foo at host alphab-bvlt-r1n89.
2017-07-21 22:06:12.941 s.m.MesosSupervisor [INFO] getMetadata: ports = [31000]
2017-07-21 22:06:13.015 s.m.MesosSupervisor [INFO] confirmAssigned: port = 31000
2017-07-21 22:06:13.017 s.m.MesosSupervisor [INFO] confirmAssigned, stack = java.lang.Throwable
	at storm.mesos.MesosSupervisor.confirmAssigned(MesosSupervisor.java:123)
	at org.apache.storm.daemon.supervisor.ReadClusterState.run(ReadClusterState.java:143)
	at org.apache.storm.event.EventManagerImp$1.run(EventManagerImp.java:54)

2017-07-21 22:06:13.017 s.m.MesosSupervisor [INFO] assigned: ports =
2017-07-21 22:06:13.017 s.m.MesosSupervisor [INFO] assigned, stack = java.lang.Throwable
	at storm.mesos.MesosSupervisor.assigned(MesosSupervisor.java:80)
	at storm.mesos.MesosSupervisor.confirmAssigned(MesosSupervisor.java:127)
	at org.apache.storm.daemon.supervisor.ReadClusterState.run(ReadClusterState.java:143)
	at org.apache.storm.event.EventManagerImp$1.run(EventManagerImp.java:54)

2017-07-21 22:06:13.020 o.a.s.d.s.Slot [WARN] SLOT alphab-bvlt-r1n89:31000 Starting in state EMPTY - assignment null
2017-07-21 22:06:13.030 o.a.s.d.s.Slot [INFO] STATE EMPTY msInState: 10 -> WAITING_FOR_BASIC_LOCALIZATION msInState: 0
2017-07-21 22:06:13.128 o.a.s.u.NimbusClient [INFO] Found leader nimbus : alphab-bvlt-r1n85:31075
2017-07-21 22:06:13.592 s.m.MesosSupervisor [INFO] SuicideDetector: _supervisorViewOfAssignedPorts = [31000], now =<PHONE_NUMBER>592, _lastTime =<PHONE_NUMBER>591

@michaelmoss : thanks for both the report and doing investigations into the root cause!  I haven't looked at the changes they made to storm-core's supervisor yet.  I understand the logic pretty well in the old clojure code (for the supervisor at least!) and I understand this storm-mesos code well.  So I'll look into it and see if it's acceptable to just accept your proposed PR for using confirmAssigned() instead of assigned().  I don't remember exactly what assigned() was used for at the moment.

Thanks, @erikdw. I'm also attempting to upgrade to 1.1.0 which I will create a separate report for (topologies will not accept any mesos resources).
This has me thinking, is there any value in this project creating separate branches for different versions of storm? How can I help?

The report you gave here about the issue in 1.0.3 is a tremendous help already!  Only way you could do more for the issue would be if you were totally confident in the behaviors such that you knew for sure that switching from assigned to just confirmAssigned would be ok (I haven't had a chance yet to look into that).
Regarding the 1.1.0 issue, if you could do a similar level of analysis on why the topologies aren't getting any slots that would be awesome.

@erikdw, cool. Will post analysis on the 1.1.0 issue.
What do you think about creating separate branches in this repo to support different versions of storm?

I really don't wanna create any more branches than we need.  We needed to create a separate branch for 1.x versus 0.x because of the package path rename (backtype.storm -> org.apache.storm).  Every extra branch means more work for backporting changes, and we are already way too thin on my team to shepherd that.

I patched this locally, by calling the 'assigned' method from the 'confirmAssigned' method which gets called at the same frequency as 1.0.2 was calling assigned. This fixes the issue and I can put in a PR for this, but I'm wondering if folks have some context as to why assigned is no longer being called and that if this change could have unintended consequences.

so... I think this might be ok.  The general behavior before:

confirmAssigned:  called by the Supervisor core to determine whether a given port it sees as assigned to this host via ZooKeeper is a port that this Supervisor should be launching a worker on.  This call is absolutely critical for the correct behavior of storm-mesos:

if you have 2 topologies running in a cluster they can run on the same host
that means 2 different topology-specific supervisors are running and trying to launch workers
only 1 of those supervisors should be allowed to try to launch a given topology's worker on a given port
so confirmAssigned is using the conveyed-via-mesos-task-launch info to say whether a port belongs to this supervisor or not
without this logic the workers would be launched randomly by any supervisor and the resource allocation and isolation would be broken

assigned: called with all of the confirmed-as-being-assigned ports

so previously we were able to put into the _supervisorViewOfAssignedPorts field all of the currently assigned ports, and that was all that this method did.
The _supervisorViewOfAssignedPorts field was only used to determine if there are any ports currently assigned, and this drove the suicide logic.

Behavior proposed for 1.0.3+:

confirmAssigned:  I'm assuming there have been no behavior changes.

Have you verified that you can run 2 topologies on the same host and that they run under the correct supervisor (and thus in the correct container)?

assigned:  unsure exactly what you passed in your call to assigned from confirmAssigned, but if we can join the 2 pieces of knowledge then I think this will suffice for replacing the purpose of the handling of assigned before:

if the port is known to the supervisor core (i.e., came in as parameter to confirmAssigned) and it's also known to have been assigned to the executor via mesos (i.e.,  _taskAssignments.confirmAssigned(port) returns true), then we have a port assigned and we can toggle a boolean to that effect.  Thus allowing us to remove _supervisorViewOfAssignedPorts in favor of that boolean.  Something like _supervisorHasAPortAssigned.
note that we might wanna move this logic to another method since this interface method isn't being called from the Supervisor core anymore.

Actually... I think this won't work:  we need to be informed or figure out when there are no workers assigned to this supervisor.  We might have to rely on only tracking the content of the _taskAssignments variable and checking that from the suicide-checker.  That should work I think.

Thanks, @erikdw. I can take a stab at this today or tomorrow. I'm assuming these changes would be backwards compatible with prior releases in the 1.0.x line, particularly 1.0.2. Will test.

Released v0.2.3 of this project with the fix from #208 included.

https://github.com/mesos/storm/releases/tag/v0.2.3

So this issue is done.

@michaelmoss : FYI, I spent some time today to dig into the storm v1.1.0+ not working issue:  it's really bad.  I file a new issue in our project here: #214


Documentation on docs.rs is a version behind

Describe the issue
The version of aws-sdk-s3 and other crates on crates.io is 0.25, but the latest versions on docs.rs is 0.24.
Links
https://crates.io/crates/aws-sdk-s3
https://docs.rs/aws-sdk-s3/latest/aws_sdk_s3/

looks like the new versions are up now


Performance Issues with sprs-ldl.

I've been using sprs-ldl to solve some symmetric sparse matrix systems and found the performance to be surprisingly poor. lsolve_csc_dense_rhs has had significantly better performance.
Here is the benchmark I've been using: https://github.com/PTNobel/sprs-performance-test
which produced the following output:
LdlNumeric took 5775ms to solve 3 sparse systems
lsolve_csc_dense_rhs took 11ms to solve 3 sparse systems

I'm happy to answer more questions if needed, but that'll probably have to come on the weekend.
This issue was filed per https://www.reddit.com/r/rust/comments/gzazna/whats_the_current_state_of_rusts_sparse_linear/fv9hf0e/?context=3

Thanks for the report @PTNobel ! Similarly to https://github.com/vbarrielle/sprs/issues/188 it would probably be nice to adapt your code to add it to the benchmarks as a starting point. I'm not familiar with the implementation so can't comment on the reasons behind this difference in performance. Someone would probably need to profile it to see what is happening..

Hello @PTNobel and thanks for the report. I'm afraid the performance of sprs-ldl is not that good right now, for several reasons:

I have not done much code-specific optimizations yet, meaning there may be some hot spots in the implementation. It would be interesting to also benchmark against https://crates.io/crates/sprs_suitesparse_ldl which is a wrapper around a state of the art Cholesky factorization.
Cholesky factorization alone is not enough for solving a sparse system efficiently, because a factorization is bound to create some fill-in which can sometimes dramatically increase the number of nonzero values. Therefore, a fill-in reduction method must be used. Currently, only reverse Cuthil-McKee ordering has been implemented (see #182 and #186), which is a start, but far from state of the art. An implementation of Approximate Minimum Degree and/or Nested Dissection would be good, but so far I've been unable to implement these. These algorithms are significantly more complex to implement and I currently don't have the time to try and implement them. Maybe binding their suitesparse implementation could be the way. It would be however interesting to see how using reverse Cuthill-McKee would fare in your benchmark. This is not yet published on crates.io though, only on the master branch.
There is a symmetry check in the algorithm, if you're confident your system matrix is symmetric this could be skipped (though the API to do that is probably inconvenient right now).

Something troubles me about your benchmark though: it is expected that lsolve_csc_dense_rhs should be 2 to 3 times faster than a solve with LDL on an already factorized system (because LDL actually does two triangular solves and a diagonal solve). However lsolve_csc_dense_rhs requires its input to be lower triangular, and does not check. So it's quite possible the solve results in your benchmark are quite different.
Another thing about the benchmark: it includes the cost to convert from triplet format to csc, but it would probably be better to avoid counting this as solve time.
I should investigate this once I'm done improving the matrix product performance (currently investigating parallelization opportunities). Please note this can take some time, I don't have much free time at the moment. Any help is welcome of course :)
Thanks for the report!

@vbarrielle, I think it sounds like the lowest hanging fruit here is to

move the csc transforms out of the timed section
Verify that the two solutions have the same results
Add the sprs_suitsparse_ldl as a test against
Apply Reverse Cuthill-McKee as another test
Incorporate the tests into the sprs benchmark framework and merge into here.
Figure out how to turn off the symmetry test

Hello, just a note that I can start investigating now that #201 is done.

Anything you want me to do?  I'm happy to try and work through my list of items from my last comment.

@PTNobel I've checked 1, and the csc transform does not take too much time so it does not matter if it stays in the timed section.
I'm going to focus on 4 and 6, which mostly requires that I publish a new version of sprs with Cuthill-McKee exposed, once it's done I can enhance the API of sprs-ldl to let the caller ask for a fill-in reduction method, and disable the symmetry check.
So if you want you can look into 2, 3 or 5.

I wanted to have comparison points with other factorization algorithms, so I serialized the first jacobian in your example and loaded it in scipy.
There I've been able to test LU factorization (scipy uses the library SuperLU under the hood):
In [23]: %timeit scipy.sparse.linalg.splu(jaco)
1 loop, best of 5: 1.9 s per loop

I've also timed the CHOLMOD algorithm, which is probably the best library for Cholesky decomposition. I've had to add a diagonal offset to avoid an error signaling the matrix is not positive definite:
In [20]: %timeit sksparse.cholmod.cholesky(jaco, 100.)
1 loop, best of 5: 462 ms per loop

For the record, on my machine, using Reverse Cuthill-McKee, I get the following timing on your benchmark:
LdlNumeric took 4379ms to solve 3 sparse systems

which would mean about 1.46s per solve but this is not an accurate comparison as the symbolic part is re-used but the backsolve is included.
In general, the number that governs the performance of the factorization is the fill-in, ie the number of additional nonzeros created in the factorization. The jacobian has 41818 nonzeros. Factoring with sprs-ldl gives 1654098 nonzeros, which can be reduced to 1498310 nonzeros with Reverse Cuthill-McKee. SuperLU has 1354237 nonzeros in its L factor, and 1354558 in its U factor. CHOLMOD has 1168635 nonzeros.
So part of the better performance in CHOLMOD is probably explained by its usage of a better fill-in reducing permutation (it probably uses AMD). It's also an extremely well tuned library.
However I find the fill-in quite big, even for CHOLMOD. But I notice the graph is wired randomly, and I'm pretty sure fill-in reducing permutations work by discovering structure in the nonzero graph, so that means this is quite a pathological example. Maybe we can expect better numbers if the graph is for instance a regular grid, or a surface mesh. That's something to try.

But I notice the graph is wired randomly

I'll add, that this is because my use-case encounters genuinely random graphs quite frequently sadly, the more structure present, the work we're doing is less useful.

So as you suspected, the results being returned are wildly different suggesting that lsolve_csc_dense_rhs was fast because it is wrong... I'll push this code to my repo tonight resolving 2.

Question:
What is the difference between LdlLongNumeric and LdlNumeric in sprs_suitesparse_ldl?

The difference comes from the type of the integer type that the underlying C library expects for the storage of the indices and indptr of the system matrix, the former expects a libc::c_long while the latter expects a libc::c_int. This mostly matters if your number of rows/columns if bigger than what can fit in an integer. Here it shouldn't matter. You probably need to call CsMat::to_other_types to convert to the correct C types.

Latest micro-optimizations in sprs-ldl (see #207) have put its performance on-par with the original C implementation of LDL (see https://github.com/PTNobel/sprs-performance-test/pull/2).
I guess the way to go now would be to have a better fill-in reducing permutation, which will require binding to those available in suitesparse, or implementing one of them.
I'll probably look into the former in the near future.

The binding of SuiteSparse's CAMD has enabled some more performance gains, and, as mentionned in https://github.com/PTNobel/sprs-performance-test/pull/2#issuecomment-706390315, I don't think there's much more improvements to be hoped for using LDL. A good way to improve performance would be to have a supernodal Cholesky decomposition (like CHOLMOD), or a multifrontal one to exploit parallelism. I think the latter is a bit more accessible than the former, so I'll probably have a look into it at some point in the future, but it's probably going to take a while, as my free time is scarce these days.

@rwl That seems like it could fit very well with this crate!

I also translated QDLDL from the OSQP solver project:
https://crates.io/crates/ldl
It is published under the Apache 2.0 license and could also be a good fit.


Convert to TypeScript

Is this really blocking?

It is if it impacts the design of the API.  90% of the work is done in #7, and I want to make sure the API design won't change as a side effect of being typescripty :)

@jkwlui @fhinkel I think I'm blocked on #2 here :)

Fixed with https://github.com/googleapis/nodejs-firestore-session/pull/7


Error when running

python port.py -t <IP_ADDRESS> -p 22
######################################################
By S.S.B Group
######################################################
Suraj Singh
Admin
S.S.B Group
<EMAIL_ADDRESS>http://bitforestinfo.blogspot.com/

Note: We Feel Proud To Be Indian

######################################################
[] IP Address Detected : <IP_ADDRESS> | Num. Of Port Detected : 1
[] Open Ports Verified.
[+] IP : <IP_ADDRESS> | Ports : ['22']
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "port.py", line 148, in banner_ip
s.connect((address, port))
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
TypeError: an integer is required

[+] Scan Started On  Tue Feb 26 08:40:26 2019
[+] Scan Finished On Tue Feb 26 08:40:26 2019
[+] Total Time Taken  0.000930070877075  Seconds

Thanks For Using My Program by SSB

At line 148 add a cast to int for port like so. s.connect((address, int(port)))

Thanks for the response,
I've created a pull request with your fix in it. Please review and accept to update the file.


[docs] - update release docs

Description of the change:
Motivation for the change:
Reviewer Checklist

[ ] Implementation matches the proposed design, or proposal is updated to match implementation
[ ] Sufficient unit test coverage
[ ] Sufficient end-to-end test coverage
[ ] Docs updated or added to /docs
[ ] Commit messages sensible and descriptive

/lgtm

/test e2e-gcp


Start using array CopyTo in tests instead of AsSpan().CopyTo

Related to: https://github.com/dotnet/corefxlab/pull/1415 where the workaround AsSpan() was added to tests.
Remove it once array CopTo is available (from this PR: https://github.com/dotnet/corefx/pull/17783)

Fixed in #1427


Mutating Attributes on Sync

Laravel Version: 5.6
Adldap2-Laravel Version: 8.0.0
PHP Version: 7.1.1
LDAP Type: ActiveDirectory

Description:
I'm trying to store and sync the "whencreated" AD attribute for users in a dateTime column in my local database.  The time in AD is in a "YYYYMMDDhhmmss.TZ" format (ex.<PHONE_NUMBER>5600.0Z), which is not a valid datetime string for MySQL.  How can I add a handler function that will mutate the provided "whencreated" attribute into the proper format?  I have the function written, I just don't know where to place it.  I'm looking at the schema docs and can't seem to figure out how.  I also need it to run when I use the sync command (Artisan::call('adldap:import',['--no-interaction']);).
Any help would be appreciated.

Hi @celticsdigital,
You will need to use an Attribute Handler.
To retrieve the LDAP users creation date, there's some built in helper methods for you:
$user->getCreatedAt(); // Returns raw created at time in AD date format
$user->getCreatedAtDate(); // Returns the created at time in a mysql formatted date.
$user->getCreatedAtTimestamp(); // Returns the created at time in a unix timestamp format.

https://github.com/Adldap2/Adldap2/blob/v8.0/src/Models/Model.php#L559-L589
Example attribute handler:
namespace App\Handlers;

use App\User as EloquentUser;
use Adldap\Models\User as LdapUser;

class LdapAttributeHandler
{
    /**
     * Synchronizes ldap attributes to the specified model.
     *
     * @param LdapUser     $ldapUser
     * @param EloquentUser $eloquentUser
     *
     * @return void
     */
    public function handle(LdapUser $ldapUser, EloquentUser $eloquentUser)
    {
        $eloquentUser->ldap_created_at = $ldapUser->getCreatedAtDate();
    }
}


[Task]: Rename BigQuery SchemaTransform providers

What needs to happen?
The naming convention for Schema transform providers should follow:
<Resource>(Read|Write)SchemaTransformProvider.
Additionally, the identifier should follow:
beam:schematransform:org.apache.beam:<resource>_(read|write):<version>.
Finally, each SchemaTransformProvider should be annotated with the @AutoService(SchemaProvider.class) annotation.
Therefore, this task should:

[ ] Rename BigQuerySchemaTransform(Read|Write)Provider to BigQuery(Read|Write)SchemaTransformProvider
[ ] Rename the configuration classes
[ ] Change the identifiers
[ ] Add the @AutoService(SchemaProvider.class) annotation

Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components

[ ] Component: Python SDK
[X] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

.take-issue


chore: remove cypress.json

Pull Request Test Coverage Report for Build 1835

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 64.286%

Totals

Change from base Build 1834:
0.0%

Covered Lines:
106

Relevant Lines:
157

💛  - Coveralls


[Feature]: Add onboarding component

Feature description
First, thank you for your amazing work on this project! It has been incredibly helpful.
I would like to request a new feature:
An Onboarding Component.
This component would be highly beneficial for new users who are interacting with a platform for the first time.
The onboarding component should guide users through the initial steps of using the application, highlighting key features and functionalities.
Features:
Step-by-Step Guidance:

[x] Allow developers to create a sequence of steps that guide users through the application's key features.
[x] Include options for both modal and tooltip styles.

Customizable Steps:

[x] Developers should be able to customize each step with text, images, or videos.

[x]Provide options to customize the look and feel to match the application's theme.

Progress Indicators:

[x] Show users their progress through the onboarding process with visual indicators.
[x ]Include the ability to skip or revisit previous steps.

Interactive Elements:

[x] Highlight interactive elements on the page (e.g., buttons, forms) as part of the onboarding process.
[x] Allow users to interact with these elements without exiting the onboarding flow.

Responsive Design:
-[x] Ensure the component is fully responsive and works seamlessly on both desktop and mobile devices.
Implementation Suggestion:
To streamline the development of this feature, we could leverage the existing popover component.
By utilizing the popover component, we can create an onboarding flow that seamlessly integrates with the current design system and functionality, ensuring consistency and ease of implementation.
Benefits:

[x] Enhances user experience by providing clear guidance for new users.
[x] Reduces the learning curve for complex applications.
[x] Helps in user retention by ensuring users understand how to use key features.

I believe this feature would add great value to the project and greatly benefit its users. Thank you for considering this request.
Example

Affected component/components
Popover
Additional Context
Additional details here...
Before submitting

[X] I've made research efforts and searched the documentation
[X] I've searched for existing issues and PRs

Friendly request
I would join you as a collaborator to this project. Please, let me know if you can invite me.

Hey @alamenai, this is a great idea. Would love to discuss further, please feel free to join our discord and reference his issue https://github.com/BelkacemYerfa/shadcn-extension/issues/58. So that everyone is on the same page. We look forward to seeing you in the discord and thanks again!

Hi @gaurangrshah , please could you share the discord link?

So sorry about that: https://discord.gg/aKcVrshP


Handle errors in switching to mobile webviews

@lizaiv77 here is a fix for mobile specifically. Please have a check. Thanks.


[Feature] Support copy-on-write for partial update merge function

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
Currently, partial update in paimon only supports mor and generates a completed changelog during full compaction. We hope that partial update will support the cow mode to directly generate complete data of columns during writing, so that downstream jobs can read the complete updated results immediately
Solution
no
Anything else?
no
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

Try changelog-producer = lookup?


Auto Refresh in General -> Action Output -> Expand xxx lines Page

Hi all,
In version 2.6 we found that in the web UI, when we click the '+ xxx more lines' link of the General -> Action Output panel, we cannot select and copy one or more lines since it seems that the page is refreshing automatically.
When we choose one line and prepare to press 'ctrl + c' the page is refreshed and what you selected has been changed.
The only way to copy that is select several lines including the line you want very quickly, then press ctrl c. Or ctrl a and ctrl c. But we may don't need to copy all of the output.

I'm unable to reproduce the problem. Can you please give more details on the browser you're using and the contents of the web console when that happens? Could it potentially be related to #500?

Hi @enykeev . I saw #500 too and yes that and this issue can be re-produced in my browser.
Stackstorm version: 2.6
Browser: Version 64.0.3282.186 (Official Build) (64-bit)
Suppose the flow's output is:
{
  "tasks": [
    {
      "state_info": null,
      "name": "first_step",
      "created_at": "2018-03-05 11:00:02",
      "updated_at": "2018-03-05 11:00:04",
      "id": "c244ec6f-3a00-41f7-95d5-6b2aabcdedeb",
      "workflow_execution_id": "3e194467-fde9-4993-b0df-8551c394318f",
      "state": "SUCCESS",
      "result": {
        "stdout": "",
        "result": {
          "status": true
        },
        "stderr": "",
        "exit_code": 0
      },
      ........
    {

Expand the action output page by clicking the '+ xxx more lines' link, the page pop up.
Try to copy several lines E.G. from  workflow_execution_id to result by select these two lines.
Frequently before you select all the lines you want, the selected and highlighted area will be changed automatically. E.G. from the first line to the current line where your cursor locates have all been selected.

@wyjuly is this still an issue in v3.4.1 or v3.5?
I'll close this at the end of the month if there has been no response.


World hider doesn’t work

Jagex made a useless update and for a week or so now the world hider no longer works. I use this plugin daily is there any way you can fix it?

Fixed in 9dac835face9203974918f2408d875c9e930cc80


Include lynnmatrix as a member

Adding myself lynnmatrix as a org member
I've  been involved in kfserving, and I am willing to keep contributing to kubeflow related projects.
Contributions:

kubeflow/kfserving#1213
SeldonIO/MLServer#92

Sponsors:

@yuzisun
@adriangonz

Output of test_org_yaml.py:
============================================================ test session starts ============================================================
platform linux -- Python 3.8.5, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/morgan/workspace/internal-acls/github-orgs
collected 1 item

test_org_yaml.py .                                                                                [100%]

============================================================= 1 passed in 0.53s =============================================================

/lgtm
/approve
Thank you for your contributions!
Welcome to Kubeflow!

/lgtm
/approve
Thank you for your contributions!
Welcome to Kubeflow!


Windows - add a DEBUG build with debugging symbols.

During the project build process on Windows, files from the official site are used. However, these files lack debug symbols, resulting in the DEBUG mode build not working. It is necessary to configure the builder to work with files from the DEBUG PHP folder, but this requires properly copying the files.

I currently don't have access to a Windows machine. Would need someone else to look into it and fix it.

Hello. Please assign this task to me

https://github.com/davidcole1340/ext-php-rs/pull/350


Integration with Vanilla enhanced.

https://www.curseforge.com/minecraft/mc-mods/vanilla-enhanced
Most of the materials I already have textures for so just make it detect the mod properly.

https://github.com/CamoMano/Enriched
Mod is now called enriched materials are:

Steel
Obsidian Alloy
Ruby
Sapphire
Tanzanite
Emerald
Copper

Merged into #15


Improve button and margin

solve issue #380

[x] format codebase using prettier
[x] make random quote button and tweet button have same right alignment
[x] fix textarea inconsistent margin
[x] recreate tweet button and fix it's hover state

@gerasaka you will have to format code properly using formatting guidelines specified in contributing.md . Then only I can review it properly.

Ok then, I'll revert the format

Looks Good!!


Remove unused function from Evergreen configuration

Copied this fix for an Evergreen error from the Java driver.

An Evergreen patch build with just a few tasks scheduled:
https://evergreen.mongodb.com/version/5c86832e3627e03aadf9e02b

Rebased and merged


Archive profiles that have no skillsets named

@romeoplat as mentioned before I'll archive them for now to assure the quality of the platform

Arkangelo Lemi (PA)
Samuel Divinda (PA)
Job Robert Sanya (PA)
Amin John (PA)
Iklas Martin (GoGirlsICT)

@Iamjaiksana maybe you can check if we can complete and reactivate some of the profiles soon?

@timmwille fir some reason I am not so pro archiving, could we just remove the belonging to some hubs for the profiles and keep them online. Cause if the hubs don't want to have them associated with them we can still Keep them since we not only have ASKnet members only


Add --database option on delete squashed migrations

Add this option required on #1467.

Coverage increased (+0.004%) to 71.293% when pulling 01c3ea94efdb1917834db45a8cdd291d45ab1432 on luchiago:feat/add-database-option-to-deleted-squashed-migrations into dba2da861b9412c1e0b67ae4b3f80df0c6bc23ea on django-extensions:master.

Coverage increased (+0.004%) to 71.293% when pulling 01c3ea94efdb1917834db45a8cdd291d45ab1432 on luchiago:feat/add-database-option-to-deleted-squashed-migrations into dba2da861b9412c1e0b67ae4b3f80df0c6bc23ea on django-extensions:master.

@trbs Sure!

(build failed for unrelated reason)

Thanks !


Could not load UVM kernel module

Ey, I got an strange error.
ubuntu@ip-172-31-2-33:~$ sudo nvidia-docker run -itd --name=tensorflow-gpu -p 8754:8888 -p 6006:6006 gcr.io/tensorflow/tensorflow:0.10.0-gpu nvidia-docker | 2016/09/25 22:24:24 Error: Could not load UVM kernel module
I put my log of commands.
log.txt

you're likely not blacklisting the nouveau drivers.
i searched your attached commands/logs for blacklist, but didn't see it.
please confirm that you followed the directions explicitly at this url:  https://github.com/fluxcapacitor/pipeline/wiki/AWS-GPU-TensorFlow-Docker
anything that strays from those instructions will be problematic.  this is a very finicky combination!  :)
lemme know.

Ok thanks! I just copied that block of commands to my shell. Thanks for your quick answer, I'll try it again :)

ah, yeah.  for whatever reason, the copy/paste doesn't work in this case.
i remember that now.
i'll break them up on the wiki.  it's annoying, i know.
On Sun, Sep 25, 2016 at 6:43 PM, EliosMolina<EMAIL_ADDRESS>wrote:

Ok thanks! I just copied that block of commands to my shell. Thanks for
your quick answer, I'll try it again :)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fluxcapacitor/pipeline/issues/124#issuecomment-249451371,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABXxcJ0kMwPk-jCl5lS4FDk6nBqz3-AOks5qtvj3gaJpZM4KGBdr
.

--
Chris Fregly
Research Scientist https://linkedin.com/in/cfregly @ PipelineIO
http://pipeline.io
Advanced Spark and TensorFlow Meetup
http://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/
San Francisco || Chicago || Washington DC

Cool now it works!


Error installing Fireant from Github

An exception is being raised
Collecting fireant[vertica] from git+https://github.com/kayak/fireant.git@master#egg=fireant[vertica] (from -r bandit/requirements.txt (line 51))
  Cloning https://github.com/kayak/fireant.git (to master) to /private/var/folders/k4/3gh_7drn1t78lh8hzlz8w6t08hdy18/T/pip-build-8g2fcnjb/fireant
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/k4/3gh_7drn1t78lh8hzlz8w6t08hdy18/T/pip-build-8g2fcnjb/fireant/setup.py", line 36, in <module>
        long_description=readme(),
      File "/private/var/folders/k4/3gh_7drn1t78lh8hzlz8w6t08hdy18/T/pip-build-8g2fcnjb/fireant/setup.py", line 12, in readme
        return f.read()
      File "/Users/mike/bin/../lib/python3.4/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 7399: ordinal not in range(128)```

I fixed this by changing the README to be plain text. RST is not parsed by pypi and the raw text was being displayed there.


How to bundle dynamic imports?

This is more of a howto, then a bug issue.
I am working on a rewrite of a vue 2 WYSIWYG editor. The editor has different "modules", which are basically the buttons and their related logic (e.g. B, I, etc.). I resolved to load these dynamically, so that I don't need to bundle every possible module.
In dev mode this is fine, but after building the import of the bundle results in this error:
ERROR  Failed to compile with 1 errors
This dependency was not found:
* @/modules in /Applications/MAMP/htdocs/hack/vue-editr/dist/vue-editr.ssr.js
To install it, you can run: npm install --save @/modules

My guess is that it cannot handle the dynamic imports here.
I do not see the modules included in the final build. How do I get that working using your setup?
Obviously, I am new to working on npm publishable SFC files.
The repo can be found here: https://github.com/digitaldeacon/vue-editr/.

This is beyond the scope of this utility, so I'm going to have to close this. You might find more helpful information in this bug report and its comments - it seems to be the same type of situation you are looking at.
Though it might not be the way you are using them here, my inclination for on-demand usage of optional features would be to export all the dynamic imports as separate, named exports of this library. Users could then import the ones they need, and pass them as options to the Vue.use or app.use (if Vue 3) command to "register" those with the plugin. This lets the user decide what to use by allowing them to inject only the pieces needed at the time of install.


⚠ eid has degraded performance

In 1f34b72, eid (https://eid.digitalservicebund.de/api/v1/health) experienced degraded performance:

HTTP code: 200
Response time: 5487 ms

Resolved: eid performance has improved in 59fc48a.


SplitButton usage question

I am trying to use SplitButton in an application and would like the following behaviors:

The content of the button (PART_Button) doesn't change when popup item from PART_Popup is changed.

The appearance of the SplitButton is similar to other types of buttons (e.g. DefaultButton) so they can be used together.

Can the current implementation of the split button do that with some customization?

@m5112638  this is more like dropdown button and not split button behavior.


Small fix necessary if you set initital-value to be your ng-model but…

… also update your ng-model as a result of making a selection. Essentially we must unbind the initial-value function in two scenarios - either if an initial-value is set, or if an item is selected.
Apologies for this one - spotted it as I was rolling out the new version across my site.

Why is this necessary?

Because if you take the initial-value as an object, the callOrAssign function is called as a result, which in turn updates your model (which is assigned as initial-value), which causes the display value to become the external model value.
It's hard to explain - but this is what I saw happening in my application, and adding this fix stopped the behaviour. I'll write a test for this asap, but I first wanted to ensure that other people didn't run into this.

When I tested it works as expected. And, unbind function should have no effect once it's called. So I don't see why we need this in callOrAssign.
If you can create an example, that would be great.

How do assign the selected value to my ng-model, looking at html code, ng-model is set to searchStr, i want to change it to my own object to save in the database. Thank you


Quick fix to address deadlock issue

CATEGORY
Choose one

[x] Bug Fix
[ ] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY
This is a fix to address an issue where FAB fails to release its underlying DB session after initializing. We are seeing an issue where locks are open to tables that need to be migrated by another process.
REVIEWERS
@mistercrunch

Codecov Report

Merging #7434 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7434      +/-   ##
==========================================
+ Coverage   65.26%   65.26%   +<.01%
==========================================
  Files         430      430
  Lines       21077    21078       +1
  Branches     2338     2338
==========================================
+ Hits        13756    13757       +1
  Misses       7205     7205
  Partials      116      116

Impacted Files
Coverage Δ

superset/__init__.py
74.24% <100%> (+0.19%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 598526a...6eafe51. Read the comment docs.

Codecov Report

Merging #7434 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7434      +/-   ##
==========================================
+ Coverage   65.26%   65.26%   +<.01%
==========================================
  Files         430      430
  Lines       21077    21078       +1
  Branches     2338     2338
==========================================
+ Hits        13756    13757       +1
  Misses       7205     7205
  Partials      116      116

Impacted Files
Coverage Δ

superset/__init__.py
74.24% <100%> (+0.19%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 598526a...6eafe51. Read the comment docs.

Awesome, thanks.


Fix issues when combining series with different intervals

When combining series with different intervals using normalize, the number of resulting values may not always be equal.  In the past the consolidating generator worked around this problem by yielding an extra None value, but that wasn't a great way to solve the problem.
Now that we require Python 2.7, we can switch from using izip to izip_longest when combining values from multiple TimeSeries, which will pad any short series with None to avoid this problem when aggregating series with a different number of values.
In _summarizeValues I also made a change to avoid trying to walk past the end of the datapoints array if for some reason it still ended up shorter than expected.
Finally, I added some test code to validate functionality with series having differing steps, as the current tests only use series with step=1.
This should solve the issue in #2113

Codecov Report

Merging #2114 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master    #2114   +/-   ##
=======================================
  Coverage   78.88%   78.88%
=======================================
  Files          78       78
  Lines        7626     7626
  Branches     1515     1515
=======================================
  Hits         6016     6016
  Misses       1383     1383
  Partials      227      227

Impacted Files
Coverage Δ

webapp/graphite/render/functions.py
95.83% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8847188...8e19e56. Read the comment docs.

Wow, cool stuff!


@conda-forge-admin, please add user @junpenglao

Additional comment:
No response

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I added user @junpenglao in conda-forge/blackjax-feedstock#4.


CAMEL-10983: Fail early and show meaningful log for invalid endpoint URI in Blueprint

https://issues.apache.org/jira/browse/CAMEL-10983

@oscerd looks good to me

Sure. Merging.

@tadayosi Thanks a lot. Can you close the PR? It has been merged.
Please resolve the JIRA ticket too :-)

Thanks!


refactor(store): change tableId type from bytes32 to uint256

This PR changes the type of tableId from bytes32 to uint256.
Reason: bytes32 clashes with string memory in the World interface because sufficiently short strings can be auto-converted to bytes32. This causes the following two method signatures to clash:
function setRecord(bytes32 tableId, bytes32[] memory key, bytes memory data) external;

function setRecord(string memory tableRoute, bytes32[] memory key, bytes memory data) external;

lgtm


Add iptables facts and modules

This branch is still work in progress, but I would like early feedback to be coherent with the rest of pyinfra.
Goal: Access iptables configuration from pyinfra ; configure packet forwarding
Checklist:

[x] Fact to list existing rules
[x] Fact to check presence of a packet forwarding rule
[x] Module to set a packet forwarding rule
[x] Tests on the facts
[ ] Tests on the module
[ ] docs.rst

In the present form, the following deploy works:
from pyinfra.modules import iptables

if not host.fact.iptables_forward(99, '<IP_ADDRESS>'):
    iptables.forward(99, '<IP_ADDRESS>')

How can I make the code from the module check for the fact ? I did not get how that is supposed to work.

I added a test on fact.iptables, but did not understand how to write tests for the module.
I tried adding a file named tests/operations/iptables.forward/present.json with as content
{
    "kwargs": {
        "to_ip": "<IP_ADDRESS>",
        "dport": 8080
    },
    "facts": {
        "iptables": {"name": "nat"}
    },
    "commands": [
        "iptables -t nat -D PREROUTING -i eth0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination <IP_ADDRESS>:8080"
    ]
}

I probably miss something in the format of the JSON test, and the errors I receive are not very explicit.

@hoh I think the fact structure is wrong - it'd need to be:
...
"facts": {
    "iptables": {
        "nat": []
    }
}
...

Updates look good - I've left a couple of other notes now that this has expanded to manage any nat rules.

Also noticed your docs update - these should/will be autogenerated via the docstring on the operation function, so you shouldn't need to manually write any docs :)

Looks like I made a mistake in the tests, just fixed it: already_absent.json and already_present.json do not return any command anymore (facts have been updated accordingly).

Function renamed, and small oversight fixed too !

@hoh thanks for the updates, all looks good now :)


Card flip

For preferences we have in NMRium the following library: https://www.npmjs.com/package/react-card-flip
However if I understand well it is buggy and when the card is flipped, in google chrome, we need to hide manually the table.
There may be a fixe in the library: https://github.com/AaronCCWong/react-card-flip/pull/61 but it was never merged.
I think we could add in this library HorizontalFlip that would do the same as this external library.

Currently this is the behaviour for the preferences.

@wadjih-bencheikh18 do you have another proposal ? Maybe without the flip.

the size of this window decrease in the direction of the (x) until it disappear will be much better


when first installing SMapp, it cannot connect to node

Environment

Smapp version: 2.1.0-beta
Os: Mac Mojave

Describe the bug

A clear and concise description of what the bug is.
app-logs.txt
node logs seems to be fine, no panics
Steps to reproduce
install the node and follow instructions

@brusherru  Smapp should allow creation of POSTs without the node being synced but the node's api obviously needs to be available. Node must be synced to create the first ATX. @noamnelke

If the node doesn't behave as expected - we need to figure out why. Can you distill the logs that Anton attached and tell me exactly what the unexpected behavior is?

@antonlerner - I'm testing now with Smapp v0.2.3 - you can create POST before the node is synced. The issue you have been seeing is likely the node being down and api unavailable.


[RFC][SGX] Use Fortanix EDP instead of rust-sgx-sdk

(WIP PR: #2885)
tl;dr: The Rust compiler now has support for an SGX target, so users can make their own enclaves using the unmodified TVM Rust runtime. In other words, TVM no longer needs to explicitly support SGX.
The current TVM SGX infrastructure is built on baidu/rust-sgx-sdk. The proposal is to replace r-s-s with fortanix/rust-sgx. Indeed, removing the rust-sgx-sdk completely obviates the need for TVM to explicitly support SGX.
Rationale
The Fortanix EDP is now a tier 3 target for Rust which vastly simplifies the build process compared to r-s-s which requires compiling a custom sysroot. Among other things, this

eliminates the need for adding SGX-specific codes to TVM (in fact, we can remove any mention of SGX from the TVM C++ and Rust codebases)
eliminates the need for adding xargo and a patched version of rust-sgx-sdk to the TVM Dockerfile
allows the use of more packages from the Rust ecosystem (e.g., the rand crate now has SGX support)
allows the use of the real Rust standard library (instead of the custom one designed by rust-sgx-sdk) and newer rustc nightlies. This benefits both correctness and security (more maintainers + reviewers).

Additionally, the EDP is a pure-Rust implementation of SGX enclaves. Compared to the Intel C++ implementation (which includes an entire C++ standard library), there is a smaller surface area of attack. That the EDP is part of the Rust compiler and is used by the Fortanix company means that the code is more actively maintained.
As shown in the updated SGX example, the EDP allows users to compile TVM modules into enclaves using nothing more than the unmodified TVM Rust runtime. Indeed, as the EDP allows running TCP servers in enclaves, all that must be done to provide a high-quality user experience for enclaves is to add TVM RPC support to the Rust runtime.
To address @tqchen's comment in #2885:

Dependency and license issue(The Fortanix one is MPL while the original one is BSD)

It actually doesn't matter because TVM under the new proposal doesn't even know that SGX exists. All we need to do is write a high-quality Rust runtime using the usual Rust toolchain.

Would the original C++ example makes it easier to port to other enclave based runtime?

Strictly speaking, the answer is "yes," but using the C++ SGX libraries is incredibly painful. For now, this does require users to write model harnesses in Rust, but we can automatically generate TVM RPC enclaves that can be called from Python.
cc @dmlc/tvm-team

No offense.
The difference between rust-sgx-sdk and Fortanix EDP roots from basic assumptions of security. Fortanix EDP is not designed for security, so it has assumptions of trusting the OS. For example, the atomicity of EDP's SGX RwLock is untrusted. Technically, Fortanix EDP maintains an event queue in untrusted space and RwLock's atomicity roots from it -- resulting in untrusted atomicity. For other input/output, they are all interacting with untrusted input/output sources.
Are you sure you want it? Every dependency crate would potentially depends on untrusted I/O and weaken your trustworthyness.

Hey @dingelish cool to see you here! Thanks a lot for the input. One point of note for the TVM use-case is that the TVM modules' thread pool already requires cooperation from the OS. In almost all (or maybe actually all?) cases, the TVM threads operate on mutually exclusive portions of the output; the operation also doesn't return until all threads have finished their tasks. Accordingly, it's not so important that the thread pool be trusted.
TVM also doesn't make use of fs nor net, so really it's just a matter of threads.
With due respect to you and your fantastic std facade (and, trust me, there's plenty of that from my end :), the Fortanix programming model is so much nicer more pleasant and flexible. That it is integrated as a Tier 3 target in rustc is no small factor in this proposal.

Yeah I 100% trust you :)
Correct me if I'm wrong: SGX_QUEUE relies on the implementation of Mutex inside SGX. In Fortanix's solution, it's relies on WaitQueue which depends on usercall to provide its atomicity. An attacker could easily hi-jack the enclave-runner thus modify the behavior of WaitQueue then gain access to the behavior of that SGX_QUEUE and further ruins the SGX environment.
A short gif to show how an attacker cheats on Fortanix's SGX app:

Well, if it's any consolation, the rustc sgx thread implementation does the exact same thing as we're already doing. Might as well reduce code duplication, right?

The code you referred to is exactly Fortanix's code. The JoinHandle is implemented using untrusted Mutex. I don't think their implementation could provide any trustworthiness. They bring too much uncertainty to the SGX environment. Personally, I strongly disagree with their implementation because they provide a LibOS-like Rust-SGX environment without any ability to control/audit the usercalls in compile time. It sounds like pushing the programmers to the edge of a cliff and say: you have the choice to not step forward. As an experienced researcher, you can hardly got away from the falling down because too much stuffs depends on their usercalls. The bad design is not desired by Fortanix, but a result of combining libstd to an environment without thread/fs/time/env/process/net. Similar runtimes such as webassembly are facing the same problem. I think you must know pwasm-std. Parity create this to provide a real runtime for wasm instead of using the default one -- you can open a file in a .rs file and compile it to webassembly, which would triggers a runtime panic. "if it builds, it works" is not true today, due to a bad design of libstd.

The current code of SGX_QUEUE (with rust-sgx-sdk) is not depending on untrusted Mutex -- it relies on sgx_spin to provide atomicity which keeps everything inside.

For others' reference, the rust-sgx-sdk Mutex uses a spinlock provided by the sgx libc.

depending on untrusted Mutex

Right, but all the untrusted OS can do is not provide threads. That only compromises availability, and the untrusted system is always able to harm availability--correctness is not affected.

LibOS-like Rust-SGX environment without any ability to control/audit the usercalls

This is a fair point. Allowing a supposedly secure enclave to trivially access untrusted functions like fs, net, and time is not a good model for users who don't understand the security implications of doing so. The approach of r-s-s which makes these modules private is certainly the more secure approach. For experienced users and library authors, however, usercalls offer greater usability.

pwasm-std

FWIW, without wasi, fs doesn't compile under wasm32-unknown-*. (sys|user)calls aside, pwasm-std unnecessarily limits itself by not including the standard library. Their focus is not security, but rather, ensuring that consensus succeeds. Of course, disallowing structures like HashMap is an oversight since their serialization is, in fact, canonical. Similarly with their disallowing floats: it's totally possible to do flops if one "simply" canonicalizes the Wasm NaN representation (or all of the miners use a single architecture). Of course, pwasm is unrelated to the matter at hand :)
As another point in favor of the fortanix edp, there's substantially lower overhead from ecalls/ocalls since it transparently implements switchless.
Overall, if the TVM runtime using more than just std::thread and those threads weren't embarrassingly parallel, I'd be more concerned about security. The main boons of switching are maintainability and usability.

Okay, it's been two weeks on this RFC, so I'm going to prepare a pros/cons summary of the discussion so far in preparation for further action.
Regarding the proposal to replace SGX support in TVM with Fortanix:
Pros

tightly integrated into Rust ecosystem, which gives support for more crates as well as a significantly simpler build process
better performance due to low overhead enclave entry/exit
better auditing of usercalls than [eo]calls; better auditing of Rust std
easier to build new functionality like RPCs
easier to debug enclaves (e.g., working backtraces, println!, panic!)

Cons

malicious operator could deny availability by blocking network access
end user needs to know that data sent outside of the enclave must go through a secure channel

I'd be glad to leave our current SGX infrastructure in place except that it adds maintenance burden as we upgrade our tvm crate. Even if the toolchain were robust enough to be tested in CI, we would still want to use Rust's own toolchain as the primary means for SGX support. Thus, unless anyone has any strong objections, we will deprecate rust-sgx-sdk in favor of x86_64-fortanix-unknown-sgx as proposed in #2885.

I'm starting a next major version of rust-sgx-sdk which could be merged into Rust's std, along with several compiler changes, new features, and lint tools. One of Pre-RFCs is here.
Would you switch back to rust-sgx-sdk v2 later? If not, we'd possibly add a similar runtime "mesatee" (maybe) to tvm to support tvm in both trustzone and sgx using our SDKs.

Out of curiosity, what will be the difference between upstreamed r-s-s and fortanix sgx? The Rust libstd implementation is already preferable to the hugely complex and large TCB Intel C/C++ libs. Are you innovating on the runner?

EDP author here. Feel free to ping me any time with any security/support questions regarding Fortanix EDP. Either on GitHub, or on Slack.

Fortanix EDP is not designed for security, so it has assumptions of trusting the OS.

Both these things are completely, entirely, 100% incorrect. (@dingelish I'd be interested to know how you came to this understanding. I'd also appreciate it if you take security concerns directly to Fortanix instead of claiming incorrect things in a public forum.)
In fact, the Fortanix EDP is designed from the ground up to be the most secure and easy to use enclave interface and platform out there.

Unlike every single other enclave platform out there, there is no C code whatsoever in the enclave.† This protects you from dangerous memory safety vulnerabilities exposing your enclave secrets.
There is only a small interface with the untrusted environment that is the same for all enclaves, such that its security can be evaluated and implementation audited once instead of for each enclave separately. The security implications of every part of this interface have been carefully designed, evaluated, and tested.
Even if you want to use raw user memory for communication, there are user reference primitives for dealing with pointers to user space. These primitives use Rust's borrowing and ownership mechanism to avoid data races and TOCTTOU issues, and also prevent creating dangerous Rust references to user memory. However, we strongly recommend you use byte streams instead, which is why Rust's core byte stream primitive (std::net::TcpStream) is made available to developers.
All code that you're required to bring into the enclave (say, to get “hello world” running) is from official Rust repositories, which receive far more scrutiny from 3rd parties than external Rust forks.

For example, the atomicity of EDP's SGX RwLock is untrusted.
...
Technically, Fortanix EDP maintains an event queue in untrusted space and RwLock's atomicity roots from it -- resulting in untrusted atomicity.
...
The JoinHandle is implemented using untrusted Mutex

Wrong. All locks are protected using in-enclave state.
Yes, there is an event queue in untrusted space. But since enclave state is the source of truth for locks, if userspace decides to ignore the event queue and wake up the wrong thread, nothing bad can happen. Since the OS is already free to schedule/deschedule enclave threads at any time, this gives you the exact same security as using spinlocks, except without the CPU overhead in case locks are held for a long time.

For other input/output, they are all interacting with untrusted input/output sources.

I don't understand your point here, SGX doesn't have trusted I/O. Everyone has to do this. We recommend you use TLS to protect all enclave communications.

they provide a LibOS-like Rust-SGX environment without any ability to control/audit the usercalls in compile time. It sounds like pushing the programmers to the edge of a cliff and say: you have the choice to not step forward. As an experienced researcher, you can hardly got away from the falling down because too much stuffs depends on their usercalls. The bad design is not desired by Fortanix, but a result of combining libstd to an environment without thread/fs/time/env/process/net.

Developing for SGX (or for any environment, really) requires the programmer to have certain background knowledge. However, there is only so much the developer can actively keep track off. This is the complexity budget. Everything else that doesn't fit in this budget needs to be reviewed and audited by others.
We've clearly chosen a different trade-off here than you. I'd say reviewing your enclave codebase for usage of std::net and letting the programmer use familiar interfaces such as HTTP REST and gRPC scores far better in terms of complexity than requiring developers to learn a new language and paradigm (EDL) for describing their enclave interface.

Con: malicious operator could deny availability by blocking network access

This is not a drawback specifically of Fortanix EDP. A malicious operator can always do this for any SGX application.
Again, feel free to ping me any time with questions.
† When using panic=abort.

NB. The target is at tier 2, not tier 3.

Looks like we've reached consensus. Thanks @dingelish and @jethrogb for the feedback. Work will now proceed on #2885.


string literals possibly being parsed as another type due to the presence of only digits and full-stops

I believe the parser is parsing IP strings like "<IP_ADDRESS>" or version strings like "2021.1.0" as something other than a string, and it fails to convert when a string is requested.
See below:
j["SW_VERSION"].get<std::string>();

Where:
"SW_VERSION":"2021.1.0"

Results in:
Exception thrown at 0x00007FFB0F0E467C in dx12win10.exe: Microsoft C++ exception: nlohmann::detail::type_error at memory location 0x000000D4FEEFEBC8.

type_name() returns "number"

Can you share a small example of your JSON and your code?


Documents how new tasking projects are setup

We will need to know how to set up new tasking projects (for once the validation process is over)

yes.  I think we have all projects up for Phase I on tasks.hothosm.org.  Yes?
Adding new Phase II projects to tasks.loggingroads.org is pretty simple.  However, linking to those projects requires some minor modifications to the jekyll site as well (enable the 'start mapping' link for that country)--really simple to do, but you have to run Jekyll locally.  I can document Jekyll setup and editing, but might be easier for me to add them when needed--wouldn't take more than 2 min to make the change.
I'm sure Rafael is willing to help as well.


⚠ ClamAV has degraded performance

In 402e712, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 72 ms

Resolved: ClamAV performance has improved in 8396331 after 5 minutes.


releaseモードのAndroidアプリでClientExceptionが発生する

実行端末はRedmi Note 9Sです。
$ flutter devices

4 connected devices:

Redmi Note 9S (mobile) • 50887fe5                             • android-arm64  • Android 11 (API 30)
iPhone 14 Pro (mobile) • 3D0DF457-008A-4A9E-8A42-F70489F0F2B2 • ios            •
com.apple.CoreSimulator.SimRuntime.iOS-16-4 (simulator)
macOS (desktop)        • macos                                • darwin-arm64   • macOS 13.4 22F66
darwin-arm64
Chrome (web)           • chrome                               • web-javascript • Google Chrome 114.0.5735.198

No wireless devices were found.

ビルド
$ makers run -d 50887fe5 --release

[cargo-make] INFO - makers 0.36.11
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: run
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: legacy-migration
[cargo-make] INFO - Running Task: runner
Deprecated. Use `dart run` instead.
[INFO] Generating build script completed, took 178ms
[INFO] Reading cached asset graph completed, took 27ms
[INFO] Checking for updates since last build completed, took 297ms
[INFO] Running build completed, took 4ms
[INFO] Caching finalized dependency graph completed, took 14ms
[INFO] Succeeded after 22ms with 0 outputs (0 actions)
[cargo-make] INFO - Running Task: run
Launching lib/main.dart on Redmi Note 9S in release mode...
Running Gradle task 'assembleRelease'...                         2,291ms
✓  Built build/app/outputs/flutter-apk/app-release.apk (7.5MB).
Installing build/app/outputs/flutter-apk/app-release.apk...      2,535ms

Flutter run key commands.
h List all available interactive commands.
c Clear the screen
q Quit (terminate the application on the device).
I/flutter (21924): ClientException with SocketException: Failed host lookup: 'api.github.com' (OS Error: No address associated with hostname, errno = 7), uri=https://api.github.com/search/repositories?q=go&page=0&per_page=30
I/flutter (21924): UnimplementedError: 検索画面でのエラー表示
I/flutter (21924): #0      Search.build.<anonymous closure> (package:yumemi_flutter_codecheck/screens/search.dart:47)
I/flutter (21924): #1      RepositoryOverviewList.build.<anonymous closure> (package:yumemi_flutter_codecheck/components/github/repository_overview_list.dart:33)
I/flutter (21924): #2      AsyncValueX.when (package:riverpod/src/common.dart:679)
I/flutter (21924): #3      RepositoryOverviewList.build (package:yumemi_flutter_codecheck/components/github/repository_overview_list.dart:29)
I/flutter (21924): #4      _ConsumerState.build (package:flutter_riverpod/src/consumer.dart:479)
I/flutter (21924): #5      StatefulElement.build (package:flutter/src/widgets/framework.dart:5198)
I/flutter (21924): #6      ConsumerStatefulElement.build (package:flutter_riverpod/src/consumer.dart:542)
I/flutter (21924): #7      ComponentElement.performRebuild (package:flutter/src/widgets/framework.dart:5086)
I/flutter (21924): #8      StatefulElement.performRebuild (package:flutter/src/widgets/framework.dart:5251)
I/flutter (21924): #9      Element.rebuild (package:flutter/src/widgets/framework.dart:4805)
I/flutter (21924): #10     BuildOwner.buildScope (package:flutter/src/widgets/framework.dart:2780)
I/flutter (21924): #11     WidgetsBinding.drawFrame (package:flutter/src/widgets/binding.dart:903)
I/flutter (21924): #12     RendererBinding._handlePersistentFrameCallback (package:flutter/src/rendering/binding.dart:358)
I/flutter (21924): #13     SchedulerBinding._invokeFrameCallback (package:flutter/src/scheduler/binding.dart:1284)
I/flutter (21924): #14     SchedulerBinding.handleDrawFrame (package:flutter/src/scheduler/binding.dart:1214)
I/flutter (21924): #15     SchedulerBinding._handleDrawFrame (package:flutter/src/scheduler/binding.dart:1072)
I/flutter (21924): #16     _invoke (dart:ui/hooks.dart:142)
I/flutter (21924): #17     PlatformDispatcher._drawFrame (dart:ui/platform_dispatcher.dart:359)
I/flutter (21924): #18     _drawFrame (dart:ui/hooks.dart:112)

AndroidManifest.xmlに権限の追加が必要です。
https://docs.flutter.dev/data-and-backend/networking


Could this be made to work also with Ikea Tradfri?

Type: Wishlist/Suggestion
Could this be made to work also with an Ikea Tradfri gateway on the network? I now realize that the difference between hue and tradfri is or was that tradfri didn't have an API for developers to tinker with but researching this a bit, it seems like people have started hacking on this topic already.
(Yes I understand that it might be out of scope for this project when it is named hue lights and not tradfri or zigbee lights.)

Hi @lwbt, thank you for your idea. Unfortunately, I suppose I will disappoint you. ....at least if I would answer right now.

Yes, it is kind of out of scope.
I do not have Tradfri gateway.

I will do some research concerning the Tradfri API/or whatever is available for it... I did not read about it yet, but I am aware there are apps that support both platforms.
Vaclav

IKEA Trafri is really popular, I have a big investment in those bulbs and gateways.


fixes issue with sfcc-ci 2.9.0 being broken

sfcc-ci 2.9.0 is broken due to a dependency issue; carat ^ resolution here will upgrade to that version when using npm install; changing to tilde ~ will keep it in the range this has been tested with (and is locked in package-lock.json)
Another possible fix is to recommend npm ci instead of npm install in the instructions to use the locked versions.
Upstream issue https://github.com/SalesforceCommerceCloud/sfcc-ci/issues/280

Hello @clavery ,
Thank you for your PR.
Can you please target the develop branch instead?
This way I'll include this change in the next release.
Thanks!

Closing to change branch

Moved here https://github.com/SalesforceCommerceCloud/b2c-crm-sync/pull/111


ICMP6 time exceeded in-transit response not sent from OVN router when source is external

We've noticed that ICMP6 time exceeded in-transit response packets are not sent from the intermediate OVN logical router hop when an MTR trace is started from an external source address targeted at an IP inside the OVN logical network (all NAT disabled).
If the source address is changed to an IP inside the same subnet as the OVN router's external port address then the OVN router responds with the expected ICMP6 time exceeded in-transit response.
Example setup:
Container connected to a bridge called "lxdbr1" with an IP of fd42:cfe0:8030:65c4:216:3eff:fe07:96a3/64.
An OVN network behind an OVN logical router, with a provider port connected to another bridge called "lxdbr0". The OVN logical router port's IP address on the "lxdbr0" subnet is fd42:6610:36a8:1234:216:3eff:fe72:513b (a different /64 than lxdbr1).
The OVN router also has a static route added to it to route fd0b:bc39:4820:d4f8::/64 to a container connected to a logical switch port. This container has the IP setup of fd0b:bc39:4820:d4f8::1/64.
On the Linux host running the two bridges, there is also a static route of fd0b:bc39:4820:d4f8::1/64 configured to route to the OVN's external IP of fd42:6610:36a8:1234:216:3eff:fe72:513b.
If I run an MTR trace from the container connected to lxdbr1, with a source address of fd42:cfe0:8030:65c4:216:3eff:fe07:96a3/64 to the OVN container's address of fd0b:bc39:4820:d4f8::1/64, then we see this result:
                            My traceroute  [v0.93]
c1 (fd42:cfe0:8030:65c4:216:3eff:fe07:96a3)           2020-12-10T12:32:21+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                      Packets               Pings
 Host                               Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. fd42:cfe0:8030:65c4::1           0.0%    11    0.2   0.2   0.1   0.2   0.0
 2. (waiting for reply)
 3. fd0b:bc39:4820:d4f8::1           0.0%    10    0.2   0.3   0.2   1.1   0.3

The OVN router in the intermediate hop is not responding with the ICMP6 time exceeded in-transit response. A tcpdump trace shows two ICMP6 echo requests, one for the OVN router and one of the target IP, but only 1 reply packet coming from the target IP, and no response from the OVN router.
sudo tcpdump -i lxdbr0 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lxdbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:34:16.659989 IP6 fd42:cfe0:8030:65c4:216:3eff:fe07:96a3 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33020, length 24
12:34:16.994012 IP6 fd42:cfe0:8030:65c4:216:3eff:fe07:96a3 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33021, length 24
12:34:16.994779 IP6 fd0b:bc39:4820:d4f8::1 > fd42:cfe0:8030:65c4:216:3eff:fe07:96a3: ICMP6, echo reply, seq 33021, length 24
12:34:17.662156 IP6 fd42:cfe0:8030:65c4:216:3eff:fe07:96a3 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33023, length 24
12:34:17.996195 IP6 fd42:cfe0:8030:65c4:216:3eff:fe07:96a3 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33024, length 24
12:34:17.996258 IP6 fd0b:bc39:4820:d4f8::1 > fd42:cfe0:8030:65c4:216:3eff:fe07:96a3: ICMP6, echo reply, seq 33024, length 24
12:34:18.664205 IP6 fd42:cfe0:8030:65c4:216:3eff:fe07:96a3 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33026, length 24
12:34:18.998078 IP6 fd42:cfe0:8030:65c4:216:3eff:fe07:96a3 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33027, length 24
12:34:18.998118 IP6 fd0b:bc39:4820:d4f8::1 > fd42:cfe0:8030:65c4:216:3eff:fe07:96a3: ICMP6, echo reply, seq 33027, length 24

However if I then setup an IPv6 NAT rule on lxdbr1 to translate the source to the host's IP on lxdbr0 (fd42:6610:36a8:1234::1) then we start seeing the expected responses:
                                     My traceroute  [v0.93]
c1 (fd42:cfe0:8030:65c4:216:3eff:fe07:96a3)                               2020-12-10T12:35:34+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                          Packets               Pings
 Host                                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. fd42:cfe0:8030:65c4::1                               0.0%    17    0.1   0.2   0.1   0.2   0.0
 2. fd42:6610:36a8:1234:216:3eff:fe72:513b               0.0%    17    1.1   1.0   0.4   1.5   0.3
 3. fd0b:bc39:4820:d4f8::1                               0.0%    16    0.2   0.2   0.1   1.1   0.2

And a tcpdump shows the expected replies:
sudo tcpdump -i lxdbr0 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lxdbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:35:52.782198 IP6 fd42:6610:36a8:1234::1 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33104, length 24
12:35:52.783417 IP6 fd42:6610:36a8:1234:216:3eff:fe72:513b > fd42:6610:36a8:1234::1: ICMP6, time exceeded in-transit for fd0b:bc39:4820:d4f8::1, length 72
12:35:53.116093 IP6 fd42:6610:36a8:1234::1 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33105, length 24
12:35:53.116787 IP6 fd0b:bc39:4820:d4f8::1 > fd42:6610:36a8:1234::1: ICMP6, echo reply, seq 33105, length 24
12:35:53.784050 IP6 fd42:6610:36a8:1234::1 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33107, length 24
12:35:53.785148 IP6 fd42:6610:36a8:1234:216:3eff:fe72:513b > fd42:6610:36a8:1234::1: ICMP6, time exceeded in-transit for fd0b:bc39:4820:d4f8::1, length 72
12:35:54.117792 IP6 fd42:6610:36a8:1234::1 > fd0b:bc39:4820:d4f8::1: ICMP6, echo request, seq 33108, length 24
12:35:54.117833 IP6 fd0b:bc39:4820:d4f8::1 > fd42:6610:36a8:1234::1: ICMP6, echo reply, seq 33108, length 24

So we were wondering is this a bug, or is this expected behaviour?

We suspect this issue is causing other recent problems around PMTU.
When the OVN network runs with a lowered PMTU, the lack of proper ICMP6 response from the gateway means that the client will not receive the needed fragmentation required response and their packets just end up getting lost whenever they exceed the OVN network MTU.
This has been an issue when providing network for ANYCAST based services as for those a common trick is to lower the MTU to 1280 which then avoids a number of issues with clients using tunnels and the services not receiving the fragmentation packets due to ECMP routing them to a different server (see https://blog.cloudflare.com/path-mtu-discovery-in-practice/)

For the PMTU part of this issue, does it make a difference to use OVN 21.09 and is the gateway_mtu option set on the Logical Router Port?
Support for generating ICMP fragmentation needed for ingress traffic was recently added and is included in 21.09.
Generation of ICMP fragmentation needed packets for traffic originating inside OVN has been there a while and requires the  gateway_mtu option to be set.

Thanks @fnordahl! I was not aware of the gateway_mtu setting as its not documented in the ovn-nb (https://manpages.ubuntu.com/manpages/focal/en/man5/ovn-nb.5.html) docs like the other options. For some reason its only mentioned in the ovn-northd docs (https://manpages.ubuntu.com/manpages/focal/man8/ovn-northd.8.html) (even though from what I can see it needs to be applied to the northbound DB like the other options). Confusing :(
Is there a recommended way to discover options such as this, I went looking for an "mtu" like setting like this first by scouring the ovn-nb manpage. In your view do you think that options such as this should be also mentioned in ovn-nb?
Thanks again.

I would indeed consider this as CMS API and as such I think it should be mentioned in the ovn-nb documentation.

I sent a patch amending the ovn-nb documentation to include this option for review<EMAIL_ADDRESS>

Amazing, thanks so much. I am testing on a later version of OVN to see if that helps with enabling PMTU.

21.09.

@fnordahl Using Ubuntu Impish with the gateway_mtu option worked well for PTMU discover (https://github.com/lxc/lxd/pull/9503).
I still need to test if that resolves the original problem reported here of no exceeded in transit message.

@fnordahl I've tested the original reported problem with Ubuntu Impish and the issue still continues I'm afraid.
No ICMP6 time exceeded in-transit response from the OVN router when packets are sent from external with a source address not in the router's external port's subnet.

Ack, thanks for checking @tomponline. At least we got the PMTU part of the problem out of the way! I'll have a look and see what could be done about the other part.

Ack, thanks for checking @tomponline. At least we got the PMTU part of the problem out of the way! I'll have a look and see what could be done about the other part.

Yes that will help @stgraber setup. Do you think that can be backported to Focal?

@tomponline that's actually a question for @fnordahl as he and his team maintain the focal packages.
In my case, I maintain OVS/OVN backports in my production PPA for my own clusters so I'll make sure I get on 21.09 there soon. The snap itself currently doesn't ship 21.09 as it wouldn't build for some reason, but I'll sort that out soon. It's also not really relevant to the snap as we only use the OVN client tools from there whereas we really need the server side updated in this case.

@tomponline that's actually a question for @fnordahl as he and his team maintain the focal packages.
In my case, I maintain OVS/OVN backports in my production PPA for my own clusters so I'll make sure I get on 21.09 there soon. The snap itself currently doesn't ship 21.09 as it wouldn't build for some reason, but I'll sort that out soon. It's also not really relevant to the snap as we only use the OVN client tools from there whereas we really need the server side updated in this case.

Yes I originally tested using the snap plus focal host and it didn't work (as expected).

We have had other parties trip over the missing PMTU from external network as noted in https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1949120, so an SRU is surely something for the OVN team in Canonical to consider. It is non-trivial though as some code re-organization took place leading up to 21.09, but we'll have a look.

We have had other parties trip over the missing PMTU from external network as noted in https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1949120, so an SRU is surely something for the OVN team in Canonical to consider. It is non-trivial though as some code re-organization took place leading up to 21.09, but we'll have a look.

Forgot to mention: On Focal an alternative is to consume the OVN 21.09 packages from the Xena UCA (i.e. set source='cloud:focal-xena' for charms or issue sudo add-apt-repository cloud-archive:xena), would that be an option here?

Xena UCA

Thanks that may be an option for, although I may also be able to use @stgraber PPA


Optimize performance of '/apps/{appId}/envs/{env}/clusters/{clusterName}/namespaces' interface queries

What's the purpose of this PR

As shown in the figure below, launching a 'findNamespaces' query will process each 'Namespace' in turn in the loop body, and when the 'Namespace ' is very large, e.g. up to 40, the total time taken is exaggerated to 7-8 seconds because of single-threaded serial execution.

Changing to multi-threaded processing of each 'Namespace' will have a very significant improvement on the 'findNamespaces' interface latency

Timing diagram from @Anilople

Codecov Report

Merging #4473 (e3eccdb) into master (afad056) will increase coverage by 0.06%.
The diff coverage is 70.58%.

@@             Coverage Diff              @@
##             master    #4473      +/-   ##
============================================
+ Coverage     53.54%   53.60%   +0.06%
- Complexity     2688     2693       +5
============================================
  Files           488      488
  Lines         15229    15238       +9
  Branches       1587     1587
============================================
+ Hits           8154     8168      +14
+ Misses         6515     6514       -1
+ Partials        560      556       -4

Impacted Files
Coverage Δ

...mework/apollo/portal/service/NamespaceService.java
68.23% <70.58%> (+1.15%)
:arrow_up:

...ervice/service/ReleaseMessageServiceWithCache.java
85.88% <0.00%> (+1.17%)
:arrow_up:

.../framework/apollo/spring/property/SpringValue.java
89.47% <0.00%> (+1.75%)
:arrow_up:

...nfigservice/service/AccessKeyServiceWithCache.java
84.69% <0.00%> (+2.04%)
:arrow_up:

...rk/apollo/spring/property/SpringValueRegistry.java
89.18% <0.00%> (+5.40%)
:arrow_up:


More moving color css to _light and non-color css, part II

Got about 25-30% done, at least for these two files.

Is there anything wrong with this?

I don't think so ... when you said 25-30% done I took it to mean don't do anything with it just yet :innocent:

Ah no. I've been trying to get others to help out with this so have been doing it in chunks.
Not a whole lot of luck on that end though >_>

Not a whole lot of luck on that end though >_>

LOL trying getting help on writing docblocks :runner: :smile_cat:


iconpack-obsidian: restrict platforms to linux

Motivation for this change
See https://github.com/NixOS/nixpkgs/pull/39941#issuecomment-387057547
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: iconpack-obsidian
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


[WHLSL] Ensure that isLValue is copied by the rewriter

Migrated from https://bugs.webkit.org/show_bug.cgi?id=189083:
At 2018-08-29T16:42:13Z<EMAIL_ADDRESS>wrote:
[WHLSL] Ensure that isLValue is copied by the rewriter

At 2018-08-29T16:59:20Z<EMAIL_ADDRESS>wrote:
Created attachment 348405
Patch

At 2018-08-29T17:02:04Z<EMAIL_ADDRESS>wrote:
Comment on attachment 348405
Patch
View in context: https://bugs.webkit.org/attachment.cgi?id=348405&action=review

Tools/WebGPUShadingLanguageRI/Test.js:275

function ternaryExpressionIsLValue(node)
{

   let isLValue;

   class TernaryExpressionVisitor extends Visitor {

       visitTernaryExpression(node)

       {

           isLValue = node.isLValue;

       }

   }

   node.visit(new TernaryExpressionVisitor());

   return isLValue;

}

This is an unfortunate test, because it doesn't describe anything user-visible. Are you sure there's no way this bug is visible without using compiler internals?

At 2018-08-29T16:43:35Z<EMAIL_ADDRESS>wrote:
Created attachment 348403
Patch

At 2018-08-29T17:05:44Z<EMAIL_ADDRESS>wrote:
Comment on attachment 348405
Patch
The last place that isLValue gets read in the interpreter is inside Checker, however after that has completed the entire program is rewritten using Inliner, which is a subclass of Rewriter.

At 2018-08-29T18:24:55Z<EMAIL_ADDRESS>wrote:
All reviewed patches have been landed.  Closing bug.

At 2018-08-29T18:24:54Z<EMAIL_ADDRESS>wrote:
Comment on attachment 348405
Patch
Clearing flags on attachment: 348405
Committed r235470: https://trac.webkit.org/changeset/235470

At 2018-08-29T18:25:38Z<EMAIL_ADDRESS>wrote:
rdar://problem/43850362


STM32H7xx: Driver for internal flash memory partially uses a fixed flash program word size, which doesn't fit for all STM32H7xx SOCs (e.g. STM32H7A3, STM32H7B0, STM32H7B3) leading to potential flash data corruption

Describe the bug
When running an adapted version of the Zephyr littlefs example on a Nucelo-H7A3ZI-Q board it fails to create and format the littlefs filesystem placed in an internal flash partition.
The UART log output shows that littlefs cannot successfully create the filesystem and fails to create superblock 0 with error code -28
(see log output further down)
Unfortunately the meaning of the error code is a bit misleading here.
Please also mention any information which could help others to understand
the problem you're facing:

What target platform are you using?
The Zephyr board name is nucleo_h7a3zi_q.
It uses an STM32H7A3ZI-Q microcontroller (ARM Cortex-M7).

What have you tried to diagnose or workaround this issue?
I tried various things to examine the issue, mainly log analysis and debugging with gdb.
Actually I think that the issue originates from the stm32h7xx specific low-level flash driver write routine flash_stm32_write_range(). More on that further down.

To Reproduce
Steps to reproduce the behavior:

Copy attached .conf and .overlay files to directory zephyr\samples\subsys\fs\littlefs\boards to add support for the nucleo_h7a3zi_q board to the littlefs example project. Remove the .txt file extension that was added because of github.
(see section 'Additional context' for attached files)
Set CONFIG_APP_WIPE_STORAGE=y in prj.conf of the example project.
This ensures, that the flash partition is always getting erased and that the littlefs filesystem is recreated.
Build example project:
west build -p -b nucleo_h7a3zi_q zephyr\samples\subsys\fs\littlefs
Connect board via USB cable and open serial console
Flash the example project
west flash
See UART log ouput, as attached to this issue, showing the error

Expected behavior
The littlefs filesystem initialization shouldn't fail and the example program should be able to execute in the intended way and do it's file operations (e.g. incrementing and writing boot counter to file). No errors should be reported in the log output by littlefs, the file system or flash driver.
File Systems API calls should work as expected.
Impact
I'm rather new to Zephyr and it took me several hours to track down and to locally fix the issue.
Initially I expected that the example project builds and would work as-is out of the box for the used Nucleo board, as support was added for it and littlefs.
So it looked like a showstopper in the beginning, but was in the end luckily more of a learning experience in debugging and working with Zephyr and about filing this bug report.
The issue does probably not only affect littlefs write operations, but potentially affects all internal flash writes on STM32H7xx SOCs  that have a flash program word size that is different from 256 bits.
Logs and console output
In the described error situation the UART log output is the following:
I: littlefs partition at /lfs1
*** Booting Zephyr OS build zephyr-v3.0.0-3700-g47850a301c41  ***
Sample program to r/w files on littlefs
Area 0 at 0x80000 on FLASH_CTRL for 524288 bytes
I: LittleFS version 2.4, disk version 2.0
I: FS at FLASH_CTRL:0x80000 is 64 0x2000-byte blocks with 512 cycle
I: sizes: rd 1 ; pr 16 ; ca 256 ; la 32
E: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1077: Corrupted dir pair at {0x0, 0x1}
W: can't mount (LFS -84); formatting
W: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1802: Superblock 0x0 has become unwritable
E: format failed (LFS -28)
E: fs mount error (-28)
FAIL: mount id 0 at /lfs1: -28

The log shows how littlefs tries to format the partition, but fails to write Superblock 0 correctly and then fails with error code -28.
Environment (please complete the following information):

OS: Windows 10
Toolchain:

GNU ARM Embedded Toolchain on Windows (gcc-arm-none-eabi-9-2020-q2-update-win32)
xpack-openocd-0.11.0-1-win32-x64
CMake 3.20.2
Ninja 1.10.2
Python 3.8.8 (AMD64 build)

Zephyr 3.0.99, Commit hash: 47850a301c419df5d01e2c5eef04ca05c48e33e1

Additional context
nucleo_h7a3zi_q.conf.txt
nucleo_h7a3zi_q.overlay.txt
Proposed fix
Change the code in zephyr\drivers\flash\flash_stm32h7x.c:376 to:
for (i = 0; i < len && i + nbytes <= len; i += nbytes, offset += nbytes) {
This essentially replaces the constant 32 by the already calculated number of bytes in nbytes.
The calculated number fits to the flash program word size of the used STM32H7xx SOC type, as it is based on a SOC type specific definition of the flash program word size.
Explanation
Original Code
...
int flash_stm32_write_range(const struct device *dev, unsigned int offset,
			    const void *data, unsigned int len)
{
	int rc = 0;
	int i, j;
	const uint8_t ndwords = FLASH_NB_32BITWORD_IN_FLASHWORD / 2;
	const uint8_t nbytes = FLASH_NB_32BITWORD_IN_FLASHWORD * 4;
	uint8_t unaligned_datas[nbytes];

	for (i = 0; i < len && i + nbytes <= len; i += nbytes, offset += nbytes) {
		rc = write_ndwords(dev, offset,
				   (const uint64_t *) data + (i >> 3),
				   ndwords);
		if (rc < 0) {
			return rc;
		}
	}
...

In the shown code part, with each execution of the for loop the number of dwords (64 bits / 8 bytes) is written to flash that is needed to match the flash program word size.

Most STM32H7xx SOCs seem to have 256 bits / 32 bytes flash program word size. This results in ndwords being equal to 32 / 8 = 4

That's the reason why the hard-coded constant 32 is used to increment the byte offset in the for loop too.

Unfortunately some (newer) STM32H7xx SOC types use only a flash program word size of 128 bits / 16 bytes (e.g. STM32H7A3, STM32H7B0, STM32H7B3 types).
As ndwords and nbytes are calculated based on definitions from the SOC type specific header file they are correct and fitting.
However the constant 32 in the for loop is not, as it would need to be set to 16 in this case.

The issue is caused by the fact, that only two dwords are written via the write_ndwords() call each loop, so 16 bytes, but the byte offset in the for loop is still incremented by 32 bytes.
When checking the flash content, one can see that a block of 16 bytes of data is followed by an erased block of 16 bytes in an alternating fashion.

Proposed solution: The code can be fixed by replacing the constant 32 by the calculated constant nbytes, as this constant respects the definition from the SOC type specific header file from the modules\hal\stm32\stm32cube\stm32h7xx\soc directory.
See the definition for FLASH_NB_32BITWORD_IN_FLASHWORD in those header files.

Log output via UART after applying the proposed solution
Littlefs detects a corrupted filesystem, as the flash partition got erased by the application.
But this time it succeeds with formatting and the freshly created filesystem can be mounted afterwards.
The file operations of the example application work without errors, too.
I: littlefs partition at /lfs1
*** Booting Zephyr OS build zephyr-v3.0.0-3700-g47850a301c41  ***
Sample program to r/w files on littlefs
Area 0 at 0x80000 on FLASH_CTRL for 524288 bytes
E: Erasing flash area ... 0
I: LittleFS version 2.4, disk version 2.0
I: FS at FLASH_CTRL:0x80000 is 64 0x2000-byte blocks with 512 cycle
I: sizes: rd 1 ; pr 16 ; ca 256 ; la 32
E: WEST_TOPDIR/modules/fs/littlefs/lfs.c:1077: Corrupted dir pair at {0x0, 0x1}
W: can't mount (LFS -84); formatting
I: /lfs1 mounted
/lfs1 mount: 0
/lfs1: bsize = 16 ; frsize = 8192 ; blocks = 64 ; bfree = 62

Listing dir /lfs1 ...
/lfs1/boot_count read count:0 (bytes: 0)
/lfs1/boot_count write new boot count 1: [wr:1]
I: Test file: /lfs1/pattern.bin not found, create one!

Thank you also for your analysis, which is very detailed, and further more exact,
And finally thanks for the fix you proposed, and the test you performed, it looks good.
So you made all the job, thus I propose you to continu the Zephyr experience, and push, on your own, the fix that you proposed in a Pull Request.

@ABOSTM
Thanks for the quick and welcoming reply. I'll try to do the PR then.
It can't be that hard to change a line of code, right? ;)


⚠ NCAA has degraded performance

In 3cb6557, NCAA (https://www.ncaa.com) experienced degraded performance:

HTTP code: 200
Response time: 1751 ms

Resolved: NCAA performance has improved in 2feae16.


General Issue: Move test SDTM datasets and programs to admiral.test package

Background Information
NOTE: admiral_ae needs to be renamed to admiralophtha_ae (so the actual program that generates the dataset needs a small update, and the dataset needs to be regenerated too) so that they have the correct name.
Definition of Done
No response

Tickets created in admiral.test. for each of the SDTMs to be updated/created.

HI @millerg23 is this all completed now? Can we remove programs/datasets from devel?

AE and OE are there (in devel branch). SC should be there very shortly, just waiting on approval. That just leaves EX, I will look to get that moved early next week.
If possible can you check AE and OE, you may need to re-install admiral.test package to get latest version.

@millerg23 Yep can see AE and OE correctly. Just one note, AELAT is missing a label in AE so maybe it should be added?


How to share a folder/file with an user/email?

Hi all,
Anyone tried to share a folder/file with another user using this SDK?  I know the model has permissions object available, I just need a small example of tying item() with permission() with share attributes to make it work if it is indeed supported..  Thanks very much in advanced.

@easy4dtam there is a create_link() attribute. You can try finding a Outlook SDK.

@easy4dtam here's the sharing docs: https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_invite?view=odsp-graph-online

Use the item.invite() method.

Thank you for reaching out and for your patience. This SDK is being officially deprecated. See #209 for more information


Error propagation in CI is broken

In the CI run https://github.com/dfinity/motoko-base/actions/runs/4983995764/jobs/8921744167#step:11:48
the TrieSet test failed (due of an unknown identifier) but the whole CI terminated with success.
This is not good.

Right; sorry about this! There was a regression in the original mo-test logic which was fixed in https://github.com/dfinity/motoko-dev-server/pull/25 about a month ago.


 I've personally tested the changes via my modifications to the Go client to support this. I would love to see it landed ASAP. We want to replace some custom logic that we had to write because this was not yet supported in the Go client.

The ASF Jenkins CI is pretty messed up at the moment. We're working on getting that fixed so that we get through the backlog of PRs.

run cpp tests
run integration tests

run cpp tests
run integration tests


/opt/intel/openvino/deployment_tools/tools/cl_compiler/clc: No such file or directory

I want to create my operator for stick2,but I find I don't
have this compiler.

Dear @QLSong
May I ask which version of OpenVino you installed ? And from where did you get the install package ? They typical OpenVino installation does not have cl_compiler under deployment_tools/tools. What I would recommend is to uninstall your existing, older OpenVino installation(s) and download the latest and greatest - which is 2019R2.01.
Thanks,
Shubha

when I reinstall the toolkit with FPGA Support, I find it.And I have created my operator for stick2 successfully, thank you very much.

Dear @QLSong
This is great. So happy you solved your problem ! Can you kindly close this ticket then ?
Thanks,
Shubha


Error handling should support debug levels

Currently error messages are either logged or discarded. It would be better to sort them in groups like informational, warning, error and failure and switch logging of them on and off with run time settings. Also grouping of errors based on subsystem (socket errors, file errors, general errors, etc) could be possible to give the application further control about the messages they will receive or ignore.

Debug levels have been added in https://github.com/lammertb/libhttp/commit/a03f2e301f1d128f41e161a3ffddd90c55ed6771
Current debug levels are:
DEBUG_LEVEL_NONE - nothing is reported
DEBUG_LEVEL_CRASH - problems affecting many connections a severe way
DEBUG_LEVEL_ERROR - problem affecting one connection in a severe way
DEBUG_LEVEL_WARNING - problem causing some minor issues on one connection
DEBUG_LEVEL_INFO - informational messages with no impact on communication quality