feat Show db total size and recycle bin size with ShowDataStmt

Proposed changes
In previous, show data stmt without use database will throw analyze exception.
After the pr show data stmt without use database will print all db size statistic
mysql> show data;
+-------+--------------------+-------+------------+-------------+-------------------+
| DbId  | DbName             | Size  | RemoteSize | RecycleSize | RecycleRemoteSize |
+-------+--------------------+-------+------------+-------------+-------------------+
| 21009 | db1                | 0     | 0          | 714         | 0                 |
| 0     | information_schema | 0     | 0          | 0           | 0                 |
| 1     | mysql              | 0     | 0          | 0           | 0                 |
| 10002 | __internal_schema  | 20105 | 0          | 0           | 0                 |
| Total | NULL               | 20105 | 0          | 714         | 0                 |
+-------+--------------------+-------+------------+-------------+-------------------+
5 rows in set (0.00 sec)
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

Pls add a regression test

run buildall

run buildall

run buildall


Files with the same name, but in different views are not followed by their path

VSCode Version: Code 1.13.1 (379d2efb5539b09112c793d3d9a413017d736f89, 2017-06-14T18:13:05.928Z)
OS Version: Darwin x64 15.6.0
Extensions:

Extension
Author
Version

EditorConfig
EditorConfig
0.9.4

vscode-cdnjs
JakeWilson
0.14.3

sort-lines
Tyriar
1.3.0

vscode-icontheme-nomo-dark
be5invis
1.3.2

theme-dark-monokai
bogdan-sinitsa
0.3.0

npm-intellisense
christian-kohler
1.3.0

path-intellisense
christian-kohler
1.4.2

vscode-eslint
dbaeumer
1.2.11

auto-close-tag
formulahendry
0.4.3

gitblame
waderyan
1.11.3

Steps to Reproduce:

open file index.html
open second file also named index.html
drag one of the files to the side to split editor

https://github.com/Microsoft/vscode/issues/13350


Rework the module toward better incapsulation and inheritability

Provide ways of overriding default classes with user-provided children on per-parser (per-document) base.
Implement common config object across all elements of the same document.
Replace CPointer classes with P6oparue
Intorduce dedicated exception classes

Along the way, tried to improve resolve-package once again by taking into account some improvents in Stash thread-safety on Rakudo side.

This is a kind of presentation PR as it is still heavily a work in progress. There is a  big problem with totally different approaches towards managing nativecasted and blessed classes. I'm trying to balance out the old box-class+box and the new view. Some things are possible no more and require appropriate replacement. For example, nativecasting done by dom-boxed trait was granting that the resulting object will be of the return type always. This is not the case now, hence a typechecking is needed. And it's only today morning that I realized the possibility of implementing the boxing via coercion.
A note on _Configurable: $!config is currently required because I would like to make sure that the config object is passed into constructors everywhere it is supposed to.
Also, if (better "when", but who knows) I finish it would be reasonable to bump the module version and, what's more important, to give it an api version. The latter is better be the same, as module's version. Say, v0.8.0.

FYI, another high volume use of LibXML is argrammon which having to import large amount of data via spreadsheetsindirectly relies on LibXML via  Spreadsheet::XLS https://raku-advent.blog/2020/12/20/day-20-a-raku-in-the-wild/

The thing which just have crossed my mind is that I haven't tested nativecasting for performance. Though internally it should  make use of class instantiation anyway, if I get the process right.
Do you think it still worth trying finishing this? Or is it better be abandoned altogether? Either way, the package resolution code from LibXML::Types worth incorporating as I added some safety and, hopefully, performance to it.

I think using CPointer classes is a significant design decision and its worth looking at this as an alternative.
It was purely done for performance reasons. but seems to be causing some maintenance issues. It's just a matter of keeping an eye on performance.
I can't get etc/benchmark.raku to run on this branch yet. But it'll hopefully give some idea on how it is performing versus  the main branch.

It would be a big disappointment if performance degradation would make it a no-go. I didn't expect this work to take as much time as it took already! Architectural-wise I do consider it better approach, but it's not the first time when speed is taking over design considerations.
Ok, let's hope I get done with my current task soon and be able to proceed further.

Some benchmarking, just to get some idea of the relative performance of new vs nativecast:
use Bench;

use LibXML::Raw;
use LibXML::Element;
use NativeCall;

class ElemClassP6 is LibXML::Element is repr('P6opaque') {
    has xmlElem:D $.raw is required;
    method box($raw) {
        $raw.Reference;
        ::?CLASS.new: :$raw
    }
}

class ElemClassCPtr is LibXML::Element is repr('CPointer') {
    method box($raw) {
        $raw.Reference;
        nativecast(::?CLASS, $raw)
    }
}

my Bench $b .= new;

$b.timethese(
    5000, {
        new-p6 => sub {
            my xmlElem:D $elem = xmlElem::New(xmlNs, 'Foo');
            for 1..10 {
                my ElemClassP6 $e .= box($elem);
            }
        },
        new-ptr => sub {
            my xmlElem:D $elem = xmlElem::New(xmlNs, 'Foo');
            for 1..10 {
                my ElemClassCPtr $e .= box($elem);
            }
        },
    });

Gives results:
Benchmark:
Timing 5000 iterations of new-p6, new-ptr...
    new-p6: 0.906 wallclock secs (0.813 usr 0.063 sys 0.875 cpu) @ 5518.429/s (n=5000)
   new-ptr: 0.430 wallclock secs (0.391 usr 0.016 sys 0.406 cpu) @ 11628.772/s (n=5000)

So it seems nativecast to a CPointer repr is  about 2x faster than new to a P6opaque repr.
That's not too much of a difference.

Depends on how much other work is to be done in between the constructions. But there be more to it. The more attributes are added the more time it takes. I'm trying to reduce that extra cost by using as little of extra attributes as possible. But, say, the whole point is to have $!config propagated from the parser down to the document hierarchy.

Sorry, I missed your early benchmarking of this on https://github.com/libxml-raku/LibXML-raku/issues/13.
Point taken, you've already optimized this PR to out-perform the current master. Please disregard the above benchmark which was performed on the current master.

I have all the chances to worsen the results. And, BTW, there were some questions to the methodology I used. So, let's not draw too hasty conclusions.
Either way, I catastrophically underestimated the amount of work necessary to get things done right. If it wouldn't be too late today when I fell like "enough is enough", I plan to commit the next intermediate state of things.

I have finally committed an intermediate state. The overall shape is now visible more clear. I'm still trying to figure out the best architecture for the construction methods, but the desire to preserve as much of the backward compatibility as possible makes it harder.
In the meanwhile, I have some considerations about the config. In order to simplify some tests I have introduced a singleton object of it which is used by construction as the default. It was solving a complexity for me where I want $!config attribute to be required until I fix all the places in code where it has to be propagated from one object to another and yet to make life easier for tests where .= new construction is used and the config as such is irrelevant.
But the more I was thinking of it the more the singleton approach appeals to me. It would let to do a couple of sensible changes to the config:

get rid of local variables and base it purely on attributes
remove most if not all use of proxies
make it more thread-friendly

With regard to the last item, it could be clearly stated in the docs that LibXML::Config.value is generally thread-unsafe. Yet, a newly created DOM by default would use a cloned copy of the global config making it totally safe for different threads to operate on different DOM trees, even if they modify their configurations in process.
What is changing is that modyfing the global wouldn't affect an already parsed document. But I consider this the right way of treating things as it reduces a risk of action at distance.

Basically, I'm done at this point. Unfortunately, there seem to be some problems with threading as 90threads.t may fail sometimes and it persistently reports weird errors about unicode. The unicode things is seemingly coming from inside the libxml2, so I barely believe I can pin down the source of problem. Beside, I've got a new task to do and would have to postpone any other activities until complete it.

I just tried to run this to look at 90threads.t. Seems to be a missinglib/LibXML/_Collection.rakumod?

Somehow it wasn't been added! Fixed.

I have located one possible source of problems and it turned out to be AttrX::Mooish which is using promises and this is a potential problem for libxml2 when OS thread change happens. So, removed it altogether and many things got seemingly in better shape. This doesn't help with malformed unicode errors though.
Finally, I tried the benchmark. Here is what I got from it.

With regard to the failing CI, it seems that LibXML::_Configurable requires at least 2022.02 to work. Most likely due to Jonathan's work on attribute initialization merged into the master somewhere in between 2021.12 and 2022.02. The problem is about $!config not getting its default value from the assignment.
Unfortunately, I don't see a viable workaround for the issue. Tried to experiment a little, but all in vain.

I consider this done, finally. The last commit:

converts LibXML::Config into a singleton-based global. Except for a couple of process-shared properties, all other are now attributes

the class is now a monitor. Though LibXML::Config.property = $value is thread-unsafe due to the method returning a container which is then being assigned into outside of monitor lock protection. So, more correct use would be either LibXML::Config.protect: { LibXML::Config.property = $value } when absolutely necessary. But otherwise the globals are to initialized as early as possible, before actually creating a parser or a document. Or a parser must receive own copy of config which would be then changed within a single thread when necessary. Alternate or additional solution would be to use LibXML::Config.set-property($value) which I consider the best alternative.
Otherwise, I'd prefer avoid using proxies as an expensive solution to getting rid of set- prefix. :)

a new configurable instance now receives the global singleton as its default configuration. This restores LibXML::Element $e .= new functionality in full. But creating a child of a DOM node still must be done either as $node.create: LibXML::Element, or as $node.box: LibXML::Element, :$raw. Where a variable exists and its type is known to be a LibXML::_Configurable the full class name can be replaced with the variable: $node.create: $elem, but it's not always possible.
This makes me think of a method which would allow to shorten the notation by accepting $node as the source of configuration object. But neither option makes me happy. I don't want to use new with some special kind of parameter (most likely named) as it should remain the low-level thing and let other classes override it the way they want it. More correct would be to use create as this is what it currently does and what it was introduced for. But then the question is how to make it as short as possible. The best I can think of so far is something like my LibXML::Element $elem .= create: :from($node); or :for($node), or, maybe, both of them. But still doesn't really feel like doing it this way.
Adding a new method doesn't appeal to me as the construction set of methods is already complete and better be left off where it is now.

Slightly unrelated but really important: I have no time left to modify the documentation. Feel terrible about it, but the job is calling and not much time left to prepare a talk for the upcoming conference.

@vrurg no hurry concentrate on your talk.
I just resynced with master including a partial fix for the unicode errors in t/90threads.t. I also improved the tests to check the validation errors are as expected. This helps to show the real issue at the test now dies,
I've got as far as golfimng the failing test. It seems to have nothing to do with threading, but shows up after running a failed relax-ng validation repeatly. The following test passes on master, but flaps, and usually dies on this branch:
use v6;
use Test;
plan 1;
use LibXML;
use LibXML::RelaxNG;

INIT my \MAX_LOOP = %*ENV<MAX_LOOP> || 50;

my LibXML:D $p .= new();

my $xml = q:to<EOF>;
<?xml version="1.0" encoding="utf-8"?>
<root><node><leaf/></node></root>
EOF

subtest 'relaxNG schema validation', {
    my $rngschema = q:to<EOF>;
    <?xml version="1.0"?>
    <r:grammar xmlns:r="http://relaxng.org/ns/structure/1.0">
      <r:start>
        <r:element name="root">
          <r:attribute name="id"/>
        </r:element>
      </r:start>
    </r:grammar>
    EOF

    my @ok = (1..1000).map: -> $n {
        my $ok = True;
        for 1..MAX_LOOP -> $m {
	    my $x = $p.parse: :string($xml);
	    try {my LibXML::RelaxNG:D $r .= new( string => $rngschema ); $r.validate( $x ) };
             with $! {
                 unless $_ ~~ X::LibXML::Parser && .message.contains('failed to validate') {
                     warn "problem after $n X $m iterations";
                     .rethrow;
                 }
             }
             else {
                $ok = False;
            }
        }
	$ok;
    }
    ok @ok.all.so, "test RNG validation errors thread safe sanity";
}

That commit fixes t/90threads.t and the golfed example above.  For RelaxNG and Schema parsing, the input buffer needs to be held for the duration of the parse, but wasn't. These buffers were being GCed during the parse leading to accessing random data, causing the UTF-8 and other random errors.

Wow, that simple...
Thinking of the convenience constructor, I think I would add the create candidate with both :for and :from available to make it possible to underscore the semantics of a call. So, my LibXML::Element:D $elem .= create: :for($doc) would mean the element is planned to be inserted into $doc DOM. and my LibXML::Element:D $elem .= create: :from($doc) would mean we plan to use it as a standalone object.

sounds good.
Some of the markdown files in docs/ were incorrect, but I think its the fault of the Makefile and  make doc
was misbehaving if Pod::To::Markdown isn't installed or any source files aren't compiling.
I fixed Makefile.in and rebuilt markdown files in docs/.
raku Build.pm6 will rebuild the Makefile.

It was a very quick and simple addition, so for now I'm fully done with the project. Giving no promises with regard to the documentation and most likely I would have to refocus on my $job tasks. But will do my best to be around and reply to any questions.

Thanks


[release-4.16] Enhance seedgen system check with additional requirement checking

Manual cherry-pick of https://github.com/openshift-kni/lifecycle-agent/pull/588.
/cc @donpenney

/override ci/prow/integration

/lgtm


Files Are Randomly Deleted

It all started after receiving the following error:
http://localhost:3000/cfs/files/books/HrqAPyftkD3nSCNzT/Resume.pdf?token= eyJhdXRoVG9rZW4iOiJISUFaUVREOUU3SUpzNTZQUGplMUY5MVk1TmEwOERvdmJGUW1STk9neEpKIn0&store=books net::ERR_INVALID_RESPONSE
And from my deduction, somehow, the files that are being stored in .meteor/local/build/programs/server/books/ are being deleted.
My code within lib/:
files = new FS.Collection("books", {
  stores: [
    new FS.Store.FileSystem("books", {
      path: 'class-books/'
    })
  ]
});

I've ran the command grep -irl "resume.pdf" .meteor and it returns only instances within .meteor/local/db/
FS.Debug = true code:
FS.HTTP.unmount:
{}
Registered HTTP method URLs:
/cfs/files/:collectionName/:id/:filename
/cfs/files/:collectionName/:id
/cfs/files/:collectionName
=> Meteor server restarted
token: eyJhdXRoVG9rZW4iOiJISUFaUVREOUU3SUpzNTZQUGplMUY5MVk1TmEwOERvdmJGUW1STk9neEpKIn0=
GET FILERECORD: HrqAPyftkD3nSCNzT
Read file "Resume.pdf" bytes 0-51036/51037
createReadStream books
createReadStreamForFileKey books

What am I doing wrong?

Having the same issue, files being deleted after meteor server restarts

I've been having issues with this, and I for me at least it seems to be related to database updates. When I pull a remove copy in, it removes all assets that aren't associated with a collection item. That's what it's looking like for me so far at least.


Model Instance and Model Program Aggregation Types

Pull Request Checklist:

[x] Positive Test Case Written by Dev

[x] Automated Testing

[ ] Sufficient User and Developer Documentation

[x] Passing Jenkins Build

[ ] Peer Code review and approval

Positive Test Case
TODO: Update Test Case (12/6/2020)
The model program and model instance aggregations are generic model aggregations types which can be used by the end-user to create specific model aggregations types (e.g., SWAT, MODFLOW, UEB etc). A standard set of metadata fields are available in model program aggregation which can be used to describe/represent a specific model program (e.g., SWAT). In addition, a metadata schema in JSON format can be used in a model program to define the metadata requirements of a corresponding model instance aggregation. When a model instance aggregation is linked to a model program aggregation using the 'executed_by' metadata field of the model instance, the metadata requirements of the model instance will be driven by the metadata schema of the linked model program. Both a standard set of metadata fields and the schema based metadata fields in model instance aggregation can be used to represent a specific model instance (e.g., SWAT).

Creating Model Program Aggregation:

Create a new composite resource. Model program aggregation can be created from a single file or a folder containing one or mode files. Upload a file to the resource. Make sure to upload a file that doesn't automatically creates an aggregation. Right click on that file and select the option "Set content type -> Model Program". Check that a model program aggregation is created. You can also create a model program aggregation from a folder. Create a new folder. Upload a file to this new folder. Now right click on the folder and select the option "Set content type -> Model Program".  You should see a new model program aggregation. You can also create a model program aggregation inside a fileset aggregation.

If you have created model program aggregation from a folder, you should be able to create other folders inside it. If you upload a file to the folder that represents model program or any of its sub-folders, that file becomes part of the model program. You can create a model program aggregation from a folder only if it has one or more files but no other aggregations inside that folder.

Editing model program aggregation metadata: In resource editing mode click any one of the model program aggregations you created. You should see metadata editing form. Try to  add metadata using this form. For the metadata field 'Model Instance Metadata JSON Schema' you can upload a JSON file that has the metadata schema for the model instance aggregation which can be linked to this model program. A model instance can be linked to a model program when editing the metadata for the model instance which is described below. Note: The JSON file you upload is not going to be part of the resource. After selecting a JSON file for upload when you save the metadata for model program, you should see the JSON schema data on the form (read only).  Anytime you want to update the JSON schema, you have to upload a new JSON file. Even though the upload JSON schema file is not part of the resource, the JSON schema will be available as a file in resource bag download or model program aggregation zip download. The naming of the json schema file is similar to how the xml files are named. The generated json schema file end with '_schema.json'. The schema json data is also written to aggregation metadata xml file.
Here is an example schema JSON file (based on SWAT model - change the extension to 'json') if you want to use: SWAT_Model_Sample_Metadata_Schema_2 .txt

Viewing model program aggregation metadata: You should be able to see all metadata that you added to the model program aggregation in resource view mode.

NOTE: Model program aggregation DOESN'T support nested aggregations - meaning a model program aggregations can not contain any other aggregation.

Creating Model Instance Aggregation:

One can create this aggregation either from a single file or a folder containing one or more files. Upload a file. Use a file that is not going to create aggregation upon upload. Right click on that file and select the option "Set content type -> Model Instance". Check that a model instance aggregation is created. You can also create a model instance aggregation from a folder. Create a new folder. Upload a file to this new folder. Now right click on the folder and select the option "Set content type -> Model Instance".  You should see a new model instance aggregation. You can also create a model instance aggregation inside a fileset aggregation. You can create a model instance aggregation from a folder only if it has one or more files including other aggregations as long as the aggregations are not of type model instance or fileset.
Editing model instance aggregation metadata: In resource editing mode click any one of the model instance aggregations you created. You should see metadata editing form. Try to  add metadata using this form. For the metadata field 'Model Program Used for Execution' select a model program aggregation from the dropdown. You should see all model program aggregations that you have view access. Where the model program aggregations are from other resources you should the resource id as part of the model program name listing in the dropdown. Once you have selected a model program, save the metadata. If the selected model program has a metadata JSON schema, you should see a button to display that schema data. If you see that button, click it to view schema data. If you didn't see this button , then select another model program for which you have uploaded a schema JSON file. If the linked model program has a metadata schema JSON file, you should see a new metadata editing form based on the schema definitions. The system generates this from dynamically based on the JSON schema. Make sure you can use this form to add metadata for the model instance. If you don't see all the metadata fields in the form, click on the top most 'Object properties' link and then check all the field names in the popup menu. Close the popup menu clicking again on the same 'Object properties' link.   Instead of linking the model instance to a model program within hydroshare, you can also specify the URL for an external model program. If you link the model instance to a model program using the URL option, then obviously you won't be able to add metadata to the model instance based on metadata schema.
Effects of schema changes to model instance metadata:

After you have added metadata to a model instance, if you change the schema by editing the linked model program, then the existing model instance metadata may or may not be anymore valid against the new schema. If the model instance metadata is invalid, you will see a message accordingly and you won't be able to make changes to the existing metadata. You will also see the existing invalid metadata in readonly mode. However, you should see a new metadata editing from based on the new schema. You can use this new form to overwrite the existing metadata.
After you have added metadata to a model instance, if you select a model program that doesn't have a metadata schema as the executed program, you should not see the schema based metadata of the model instance in editing mode. However, you should be able to see this metadata in view mode. Also this schema based metadata is always written to aggregation metadata xml file even if the metadata is not valid. All schema based metadata xml encoding is based on namespace 'hsterms'.

Nested Aggregation Support in Model Instance Aggregation.  A folder based model instance aggregation can contain any other aggregation types except model instance or fileset type aggregation. Similar to fileset, auto aggregation creation is supported inside a model instance aggregation. For example , if you upload a tif or nc file into a folder that represents a model instance aggregation, you should see a new raster or multi-dimensional aggregation gets created. You can also move another allowed aggregation type into a model instance aggregation folder.

Coverage Propagation: The way the coverage propagation from the contained aggregations to the parent model instance aggregation works is similar to the fileset.

Aggregation name change: You should be able to change the filename of a single file based model instance/program aggregation. You can also rename the folder that represents  a model instance/program aggregation.

Download Model Program/Instance:  Right click any model program or model instance aggregation and then select 'Download zipped'. Unzip the downloaded file and verify the contents of this zip file. In addition to the 2 xml files you should also find one json file in case of model program if the model program contains metadata schema. Do the same check by downloading the resource bag.

Removing Model Program/Instance aggregation: This should work same as for any other aggregations types.

Publishing Requirements for Resource Containing Model Instance: If a model instance is linked to a model program aggregation from another resource, then the resource containing the model instance can be published only if the other resource has already been published. If the model instance is linked to an external model program (using the URL option), then the resource containing model instance can be published  only if the external model program referenced URL is a DOI.

I haven't forgot about this @pkdash, just a bit overwhelming to take on and need a large enough chunk of time.

@sblack-usu This is NOT ready for review. I created this PR to basically run the tests. I will let you know when it is ready for review. Thanks.

noted.  This work is already very large, we should have some reviews happening soon even if you're not completely done.

@pkdash - In case you didn't see my earlier comment, I'm holding off on reviewing this until the bad merge conflict resolution is resolved.

@pkdash - looks like the bad merge conflict resolution has been fixed yet.  Do you need anything from me to fix it?  I'd like to start reviewing this week if possible.

METRIC
VALUE

NEW DUPLICATED LINES DENSITY
7.12171339021417

NEW VULNERABILITIES
0

NEW SECURITY HOTSPOTS
2

NEW CODE SMELLS
54

NEW BUGS
0

https://sonarqube.cuahsi-workstation.com:9000/dashboard?id=hydroshare-3792

@sblack-usu I have now fixed the bad merge issue. But not sure if I did it the right way. Let me know.

Thanks @pkdash, looking forward to reviewing!

Not done with the review yet.  I've looked over all the backend changes and have voiced my thoughts.  I'd like to spend a few days refactoring some of the code organization before @pkdash tries to address my comments too much.  I'll be working on this tomorrow as well as the first part of next week.

@sblack-usu I was aware of the conflicts.  It is taking quite a bit of time for me to make the model aggregations ingestion to work. I did not realize you have updated this branch with develop until now when I tried to push code changes. I guess I will resume my work after I update my local branch next weekend.

@pkdash - Sorry, I didn't realize you had changes locally that hadn't been pushed yet.  Feel free to force push your local branch and then I can work with and look at that.  I had something come up while working on the conflicts so I haven't been able to pick this up again until now.

@pkdash I don't see my comments being posted by github after submitting my comment, so here it goes again. I have finished my first round of code review for this big PR with a lot of comments. Scott has good comments that I don't see being addressed yet, so let us know when it is ready for another round of code review. I am trying to fix that discover api pytest failure, but haven't been able to fix it yet. I'll dig into it further trying to fix the test failure, though.

I'll dig into it further trying to fix the test failure.

The reason for the test failure was a fundamental change made to SOLR in an older PR. @alvacouch can explain the reasononing for the change. The impact to this test is just to add a line. I'll open a PR into this PR.

@sblack-usu and @hyi Sorry by mistake I requested review. I am still working to address your comments.

Congrats @pkdash - this is going to beta today.

Will follow up with moving the pip requirements to hs_docker_base soon.

@sblack-usu and @hyi Thanks for your help on reviewing such a large PR with many constructive suggestions for improvement. Last few months have been difficult to find time to work on Hydroshare.


[NG] tests are added and updated

Signed-off-by: stsogoo<EMAIL_ADDRESS>

The tests themselves look ok to me. I didn't review the components and services and will need some more time to look at this in context of these tests.


Fix CORS problem on Flutter Web

Fix the following error on browsers
Access to XMLHttpRequest at 'https://api.mixpanel.com/track/?data=...&verbose=1&ip=0' from origin 'https://some.domain' has been blocked by CORS policy: Request header field content-type is not allowed by Access-Control-Allow-Headers in preflight response.

Changes:
Remove 'Content-type' header from GET requests

This should fix problem described in #5

Hi @HaoCherHong thanks for this PR. I've tested it and it seems to be working! 🙌🏻


KeyError: 'standard_id' on traverse all implicit relationships

Running the "Malware to all Entities" relationship on demo.opencti.io, using Emotet (malware--36297121-bf18-56e1-b200-d39c9f9f5cee) as input:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/maltego_trx/server.py", line 45, in run_transform
    return transform_method.run_transform(client_msg), 200  # Transform class
  File "/usr/local/lib/python3.8/site-packages/maltego_trx/transform.py", line 12, in run_transform
    cls.create_entities(request, response)
  File "/var/www/TRX/transforms/openctimalwareStixDomainEntityToStixDomainEntity.py", line 12, in create_entities
    opencti_transform("StixDomainEntityToStixDomainEntity", None, request, response)
  File "/var/www/TRX/opencti/openctitransform.py", line 273, in opencti_transform
    neighbour_entity = addStixEntity(
  File "/var/www/TRX/opencti/addEntities.py", line 38, in addStixEntity
    if opencti_entity["standard_id"].startswith("identity--"):
KeyError: 'standard_id'

This of course makes the Transform fail.
In other news though, the grouped API calls are hugely speeding up the integration, very awesome!

Thanks for the tests.
I wasn't testing with the open demo so this case was not obvious on my side my I had several issues when Observables were directly linked to Domain Entities.
The errors should be gone, but I'm still investigating a bug on OpenCTI side, where relations are not properly filtered when I query for "Stix-Domain-Object" only on one side. In the current version, it may just return less info than 2*limit because some relations are filtered after the query.

Waiting on https://github.com/OpenCTI-Platform/client-python/issues/158


Completely unstable MC flight v1.4.1rc2

I had successful flight with tag v1.4.1rc1 logs here (intensive wind conditions)
But when I switched to tag v1.4.1rc2 or master (at today's commit eda2915f0b4706ddfbd6df9770ce237baa5c5ad9) I have following problems:

I experience the same issue as her #5029.
It get GPS 3d fix but doesn't go ALT hold, unless you put LPE_GPS_ON = 0
a copter is completely unstable - barely hold the plane attitude indoor and falls down after few seconds outside with wind. Estimator - LPE, all the PIDs are the same as the were for excellent performance flights. Some falling logs: http://logs.uaventure.com/view/EQAsb8tZDSbbsrKTMDhZzB
http://logs.uaventure.com/view/VPVTTbxNrf5TWXn7PLnHdD

Thanks for reporting. I'm assuming the error was due to #5029 and fixed with #5047. @bkueng am I right?

@julianoes yes the stability issues should be gone. Some more testing would be welcome, just to be sure. There's still the calibration issue Gyro #0 failure : Stale data!. Could you look into that? It seems that after calibration, the gyro 0 doesn't return any data for a while and thus the voter fails over to the next gyro with a timeout message.

@bkueng strange. I've seen similar issues on Snappy before where I would see baro timeout from the commander (which hints at a slow sensor update) right at the end of the calibration.

@bkueng, @julianoes  In addition to the message Gyro #0 failure : Stale data!, I also get a message, that in preflight checks Gyro0 failed. Does Gyro calibration work at all? With latest master, I get I continuously increasing pitch offset, which is hardly to compensate by RC after a couple of seconds. Recalibration did not fix this.

@ecmnet Hm yes, something seems wrong. I'm looking into it as well.

Thank's! Will do more tests on Monday!

@julianoes I did some more flight tests with the latest master. Still I get an increasing pitch drift with it. Reverting to 1.3.4 solves that issue.

@ecmnet: can you share a log where that happens?

@julianoes Here is one: http://logs.uaventure.com/view/GwPvYyz3iR5vEsiEpoVmPh

@ecmnet I had a look at the log but I don't know how I could see or tell the pitch drift.
Does it also happen when you're not flying but the quad is just sitting on the table? If yes, can you try to arm (e.g. without props) and without auto-disarm set and leave it for some minutes?

@julianoes If you compare  http://logs.uaventure.com/view/GwPvYyz3iR5vEsiEpoVmPh#AS_Pitch_PLOT with the radio input shown in http://logs.uaventure.com/view/GwPvYyz3iR5vEsiEpoVmPh#RC_Raw_PLOT (blue line for pitch) you can see that i had to correct pitch massively to keep the vehicle at its position. It as also very obvious, that the effect was increasing over the time. After a couple of seconds I was hardly able to control the vehicle at all.
Currently I have 1.3.4 on my quad which works fine. At the end of the week i could be able to test master again.

I did tests yesterday with v1.4.1rc1 and v1.4.1rc2 master. The initial issue was solved at master so the copter was capable to flight, but the performance is significantly worse than v1.4.1rc1.
Here are the logs of the same day with the same params:
v1.4.1rc1 - http://logs.uaventure.com/view/Y8gCg2BVo6bkpwGermLLcN
v1.4.1rc2 master - http://logs.uaventure.com/view/Dv85xGWkRmqnaeyQptb33Z
On the latter one the copter is much more twitchy, holding the XYZ position much worse.
I am also getting INFO messages like these:
2016_07_19_15_42_13: [lpe] gps fault   2   0  23   0   0   0
2016_07_19_15_42_17: [lpe] gps fault   2   0  15   0   0   0
flying rc1 firmware I might get 2-3 of these messages during the 15 min flight, while flying rc2 I get around 100 of these messages.
It doesn't look like a hardware issue, since I tried to change GPS modules to different vendors and the issue remains.

The pitch issue might be connected with #4879.

The pitch issue might be connected with #4879.

I don't think it's connected to the integral gain of the controller because the control performance looks decent looking at ATT.Pitch vs. ATSP.PitchSP.

@ilia-sheremet Thanks for reporting.  Comparing the Gyro data between 1.3.4 and master, I see a significant increase on noise. (1.3.4 <+/-0.05 vs. master <+/-0.3). Any idea where it could come from?

@julianoes I made some comparison with the quad on the ground. Procedure: arming and then increasing throttle (just not taking off).
GyroY 1.3.4 =>

GyroY Master 8ab841e0467d8509db9f97019c640ea8da2cdaff =>

Pitch 1.3.4 =>

Pitch Master 8ab841e0467d8509db9f97019c640ea8da2cdaff =>

There seems to be a significant difference in scale.

@ecmnet Interesting, thanks for the data! Can you tell me what Hardware you are using? Pixhwak, Pixfalcon, Pixracer? And by any chance, do you know which gyro is used, either the MPU6000 or L3DG20.
You could also compare the values doing:
mpu6000 test
l3gd20 test

@julianoes I am using Pixhawk. Both running with standard setup (LPE). Without arming, I get
1.3.4 =>
mpu6000: gyro y:          0.00346       rad/s
l3gd20: gyro y:               0.00255       rad/s
Master =>
WARN  [mpu6000] gyro y:           0.00267       rad/s
WARN  [l3gd20] gyro y:              -0.00223       rad/s
Besides the sign for L3GD20 in master, quite similar.
How can I figure out which gyro is used? The data shown above is just MAVLink published data.

@ecmnet I think you're just looking at noise if you just look at one instant.
If you're on a Pixhawk and you're not getting any messages like "gyro 0 fail, failover", it should be using 0 which is the one that is started first in https://github.com/PX4/Firmware/blob/master/ROMFS/px4fmu_common/init.d/rc.sensors#L94, so the MPU6000.
Let me try to reproduce what you found.

@ecmnet I compared the two versions on my Pixhawk and I can't see a difference, both gyro calibrations work as expected and the attitude estimate is decent.
If you see a very clear difference, could you try to do a git bisect between the two versions to see which commit causes the troubles?

@julianoes The difference in attitude estimation and gyro is cleary visible above. I'll try to figure out, which commit seems to be responsible for this effect. I understood, that you were not be able to reproduce my issue. Did you do the kind of throttle-ramp-up, I did? The strange thing is, that when running the ash tests, the magnitude of 1.3.4 and master is the same. Looking at the published MAVLink-Data - even when not armed -, there seems to be a difference by nearly half an magnitude (1.3.4: about 0.02 vs. master about 0.07), if armed this get worse.

@ecmnet I just have some general comments for you not comparing different versions.
In most of the logs I have seen above the IMU data is all over the place, indicating that your vibration damping is not optimal. Particularly here you can see that LPE diverged and did a reset http://logs.uaventure.com/logmuncher/charts/GwPvYyz3iR5vEsiEpoVmPh/Local_X.png
Additionally, the attitude estimator q uses the estimated velocities form the state estimator to compensate for non-gravitational acceleration. So if your velocity estimation diverges or is bad I can well imagine that your attitude will also suffer from this.
I think you need to start fixing your setup!

@tumbili I totally agree with you and I am aware of the vibrations already.
Nevertheless, I am curious about the different behavior between versions, which I try to understand.
So my basic question is, what has changed. LPOS, in my case, is based on PX4Flow data only and has continuously timeouts, etc - those should - in my opinion - not affect the attitude at all. Was the compensation of acceleration based on the state estimations introduced shortly?

@ecmnet I'm not exactly sure what's the cause of it but there are several possibilities.
Having a brief look at the last logs from @ilia-sheremet I just notices than in his old logs the range of acceleration seems less. I'm also not sure if between those versions something in the IMU driver code changed (e.g. filtering). Also changes to the estimator parameters could cause this, you would have to sync up on this with @jgoppert. If the LPE would log estimator innovations then that would make our life a bit easier analyzing the issue, see https://github.com/PX4/Firmware/issues/4762

@tumbili looking at the ekf2_innovations message. What is vel_pos_innov? Is that just for gps measurement?

@jgoppert those are generic innovations for NED velocity and position. At the moment ekf2 does not fuse several measurement sources for the same axis.
So e.g. if you are using baro to estimate height, then vel_pos_innov[5] will be the baro innovation, however, if you are using a range finder for height estimation then it will be the innovation for the range finder.

Just created a PR #5124 to add innovation logging to LPE, so this may help diagnose this.

@jgoppert @tumbili @julianoes This might help during flight, but I doubt, that this would help to explain the effects on ground, I've shown above. Meanwhile I did some simple vibration damping, and was able to decrease gyro noise by about 30%. Again I did a on-ground-comparison between 1.3.4 and master - and again, I saw pitch increasing over the time on ground. Flight shows again the effect: Only slow drift with 1.3.4 and again increasing pitch drift with master.

@julianoes Just tested 1.4.1 RC1. This version works fine for me.

@ecmnet Please share the latest log where pitch drifts

@tumbili I posted one 4 days ago: : http://logs.uaventure.com/view/GwPvYyz3iR5vEsiEpoVmPh

@ecmnet Which one is the one on the ground?

@tumbili For the ground test, I've no logs, but published the screenshots above.

@ecmnet I need those logs :-)

@tumbili Try to find them...did a lot tests today

@ecmnet In those plots were you actually moving the quad or was it just sitting on the ground?

@jgoppert does your PR suppose to solve the issue I mentioned regarding [lpe] gps fault ?

@ilia-sheremet No, it's only to help us analyze the problem.

@ecmnet I also see that your max gyro bias for the attitude estimator is set to zero.

@tumbili With RC2 the drift appears again. Here's the log while on the ground, arming and increasing throttle: http://logs.uaventure.com/view/kARbQ4oPptvvpSX3nzz9oS

@tumbili Is that a new parameter? Does that mean, that gyro bias is not compensated?

@ecmnet No it's not new at all. And yes, I guess that's what it means.

@tumbili Will make another log then with this PR.

@tumbili Same test on ground again with 1.4.1RC1 for comparison: http://logs.uaventure.com/view/wuC4PJ5CnvVKKHaW9nGmi4

@ecmnet I compared the two logs and performed a manual integration of the rates to see if they correlate with the estimated roll and pitch angles. It really seems like the older version gets you better results. I scrolled through the commits and the biggest changes between the two versions is the refactoring of the sensor topics @bkueng FYI

@tumbili thanks a lot for your effort analyzing the data. Do you have some measurements/logs to benchmark the setup?

A possible cause of sensor value changes that I see with my sensors refactoring is: the raw sensor topic has both the values & the integral. After the refactoring, if the integral_dt is set, the integral value will be used (prefered choice), otherwise the value is used. So you can check what has been used before (I think it was the integral as well), and whether using the values improves things.

@bkueng I believe that before the values were used not the integral  but I need to check. In theory it should result in the same thing but who knows...

@jgoppert Just made a log with your PR.
Here is the link to uaventure log, or the px4log itself and message files. Apart from GPS there was also a baro fault message.
There was quite intensive wind around 8m/s with 10-11 gusts, the copter is also small around 500 class.

@tumbili @bkueng The pitch issue appears also in V1.4.1RC4.

@ecmnet do you have a logfile for the last test? Is this with Pixracer or Pixhawk? How is it dampened?

@julianoes Please find the log attached. This is with Pixhawk. I am already aware that damping is not optimal with my setup, but this issue occurs only from 1.4.1 RC2 on; RC1 works fine for me.
RC4 log: http://logs.uaventure.com/view/YwZWjefMWDd56KaLW6iBy8
RC1 log:  http://logs.uaventure.com/view/w6HFPnTK8MkeQA3HJYg6J5
As mentioned by @bkueng the sensor logic was changed between those RCs.

@ecmnet Thanks for your tests. Just to be sure it's the sensors logic, can you change the line
https://github.com/PX4/Firmware/blob/master/src/modules/sensors/sensors.cpp#L1161
into:
	if (false) {

And the same for the accel_poll method?

@bkueng @julianoes I just did some short tests changing sensors.cpp not to use the integrals.

Test: Changing gyro polling only: Seems to be better, but still some drift.
Test: Changing both, gyro and acc: The issue is fixed. Quite stable flight without any significant drift.

@julianoes I'll do a test certainly. Perhaps you could give me some short explanation why this could be connected to the integration issue of sensor.cpp? As far as I can see you added a low pass filter to the attitude estimator. This would apply to both methods of sensor.cpp, doesn't it?

The nature of the complimentary filter tuning is that it expects a certain conditioning of the input data and the frequencies contained in it. We achieve this with the filter.

I can't see the addition of a complimentary filter in this change, but a low pass filter for acc and gyro data: https://github.com/PX4/Firmware/pull/5216/commits/1c683ea13556ce9e92a308533473e77c7344c0d3
As I faced a kind of bias as a result of the attitude estimation if using the gyro/acc integral divided by integral_dt instead of the reported value itself , my question is, how a low pass filter can help here, if it is applied in a later step. Anyway, I'll try to give it a test today.

You need to understand the frequency response of attitude_estimator_q, which is a complimentary filter, to understand this change.

@ecmnet before the change in sensors.cpp, the filtered gyro/accel values were fed to the attitude estimator q, however with the change we're trying to move to the unfiltered but integrated sensor values.

@julianoes Ok. Got that. I was not aware of the fact that the gyro/acc values were already filtered in the driver, but the integrals were not. I'll share the log here as soon as I've performed the tests.

@julianoes  @bkueng @LorenzMeier Did a short flight test with branch fix_q_filtering. That change did the trick: No significant pitch drift anymore.
Log:  http://logs.uaventure.com/view/EWSVDvwceuywRHSzgDhSbN

@ecmnet thanks for the quick test turnaround, that's good news!

I flew this morning with the latest master (more details in my last comment here). I didn't notice any more twitchy moves due to the messages like these:

2016_08_04_11_05_42: [lpe] gps fault   0   0  49   0   0   0
2016_08_04_11_06_55: [lpe] gps fault   0   0  28   0   0   0
But I still was getting the messages during the flight, around 20 per 10 minutes flight. Getting more of them while moving than holding the position.
I use quite updated Drotek GPS module with a big antenna, so there shouldn't be hardware issues.
So why these messages appears? What do they signify?

@ilia-sheremet The message is the gps residual value, error from prediction  (x, y, z, vx, vy, vz, rouneded to nearest m, m/s respectively) So your gps reading was off briefly 50 m from the estimated altitude. I get this some times when the vehicle leans over a lot and the gps patch antenna is getting poor reception.

@jgoppert Thank's! It was actually leaning quite a lot during the flight due to the gusts.
All the messages I got were indicating errors in Z plane exclusively, so I believe it's normal behavior?

@ilia-sheremet Yes,  it is normal. I have seen it on various ublox models.

This and the more twitchy response will lead to user comfort issue reports - should we apply two bars for this warning and count the first into a sum and only report the higher bar?
Same is true for the control response: on a racer the current tuning leads to audible 5 Hz twitches as the GPS updates occur

This is resolved on master, yes?

@LorenzMeier Yes, it's solved.


"Current PCRs" cannot be locked until next Caliptra warm reset

This leaves these PCRs open to erasure by runtime firmware. Hardware PCRs that are freely erasable by firmware do not have sufficient value as a store of measurements.
Either RTL should support an extra warm reset domain for PCR locking, or "current PCRs" should be retired as a concept.

"Lock" will not allow FW to clear. Lock will still allow you to extend the PCR. Its already in the RTL

What I mean is that a PCR can only either be locked or unlocked, there's no in between.

Locked: cannot be cleared until a power cycle.
Unlocked: can freely be erased by runtime firmware.

An unlocked PCR is not a safe place to put runtime firmware measurements if you're worried about runtime firmware lying to you. And a locked PCR is only suitable for a "journey" PCR, not a "current PCR".

If current is erased, because PCR signing is done by HW with the key that is 'locked' by ROM - what is present in PCRs is what is signed! So even FW lied, the signing cannot be fooled

But FW can replace the contents of "current" PCR with whatever FW wishes, then allow HW to sign the FW-controlled PCR contents. So we have ROM and FMC extending measurements into a "journey" PCR, with the assumption that HW-generated quotes will contain an accurate reflection of the measurements that ROM and FMC actually captured. But there's no guarantee of that. FW is free to replace the measurements that ROM and FMC extended.
So even though HW can emit an honest quote of the value of these PCRs, no one should be making security decisions about those unlocked PCRs' values if they do not already trust FW.

I dont understand that statement - why do we do PCR signing and report PCRs if no one is there on the other side to make decisions on it?
Independent of what Caliptra is doing - a man-in-the-middle can also 'erase'/modify the 'current' PCRs.
I am probably not understanding your view - plz let me know

Just to summarize the offline discussion - PCRs that can be reset by Caliptra FW are not trustworthy places to hold measurements of Caliptra FW.

Fixed the bug of moving the PCR block to be on core reset/FW update reset domain to account for the case where "RT FW PCR" can also be unlocked/locked. this came up on a different thread.

Does that mean that the PCRs cannot maintain state across hitless FW updates? Or does it just mean that all PCRs get autonomously unlocked across hitless FW updates, but their value remains the same?

They can - only PCR lock is being reset so that it can be locked!

Excellent, thanks!


Paper Writing

@BostonTLee I've been looking into coverage probabilities a little bit and this paper and this paper seem to be able to define the coverage error in terms of O(n) but not explicitly. Let me know if you see something different but I think they're saying that we can describe the behavior of the coverage but not actually estimate it. Their proofs are really difficult to understand, though.

Alright, so a couple of things.
First, about content changes:

To make figures and tables work, I changed to the bookdown package. You will likely have to install it.
The bookdown package will let us reference the figures with \ref{fig:[chunk name]} and tables \ref{tab:[chunk name]}. This seems tedious, but it auto-labels all figures and tables for us, so we won't ever have to worry about that.
The formatting is wonky right now because we have more figures than writing. This should get better once we add a results section proper.

Second, about organization. I think we should give a "Methods" overview of what we planned on doing (bootstrapping, etc), and then a "Results" section where we describe the specific results and their meaning with figures, tables, etc. I made the figures, but there is no writing in "Results" right now.


added RawArgs to Context

See issue #25.

Sorry for the long wait. Haven't had enough time to look at this. I plan to review soon.

+1 on this :)

Sorry I had to close this PR.
Some new features lead to this PR becoming outdated. Coupled with the fact that I wanted to request changes on this PR.
I went for the faster approach and added the implementation after some new fixes I did. You can check it here https://github.com/abiosoft/ishell/commit/9031de3d2f5bea9713b1e8227ab3f60c71fb94d2.
Thanks for spending time on this. And still looking forward to you contributing (again).


Android - recorded WAV file cannot play with ExoPlayer

In Flutter, same error as below.

https://github.com/ryanheise/just_audio (baked android exoplayer)
https://github.com/florent37/Flutter-AssetsAudioPlayer  (baked android exoplayer)
https://github.com/danielR2001/flutter_exoplayer (baked android exoplayer)

https://github.com/ryanheise/just_audio/issues/158
E/ExoPlayerImplInternal(12865): Source error
E/ExoPlayerImplInternal(12865):   com.google.android.exoplayer2.ParserException: Expected block size: 2; got: 1
E/ExoPlayerImplInternal(12865):       at com.google.android.exoplayer2.extractor.wav.WavExtractor$PassthroughOutputWriter.<init>(WavExtractor.java:222)
E/ExoPlayerImplInternal(12865):       at com.google.android.exoplayer2.extractor.wav.WavExtractor.read(WavExtractor.java:122)
E/ExoPlayerImplInternal(12865):       at com.google.android.exoplayer2.source.ProgressiveMediaPeriod$ExtractingLoadable.load(ProgressiveMediaPeriod.java:982)
E/ExoPlayerImplInternal(12865):       at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:391)
E/ExoPlayerImplInternal(12865):       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
E/ExoPlayerImplInternal(12865):       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
E/ExoPlayerImplInternal(12865):       at java.lang.Thread.run(Thread.java:764)

#34
#11
In the #11, has some workwround.


fix(views): correct translations not added in weekend meeting

Summary by CodeRabbit

New Features

Introduced language localization for various roles within the Weekend Meeting component, enhancing user experience for non-English speakers.

:tada: This PR is included in version 2.123.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Dlib exception_ptr error while cross-compiling for arm

Hello, I am  facing a issue here. I have cross-compiled opencv for arm using a cross-compilation tool-chain kit. I included DCMAKE_CXX_FLAGS="-std=c++11" to compile opencv and it works. However, when i use the same compiler to cross-compile dlib it is giving me error like:
Scanning dependencies of target dlib
[  0%] Building CXX object dlib/CMakeFiles/dlib.dir/base64/base64_kernel_1.cpp.o
[  1%] Building CXX object dlib/CMakeFiles/dlib.dir/bigint/bigint_kernel_1.cpp.o
[  2%] Building CXX object dlib/CMakeFiles/dlib.dir/bigint/bigint_kernel_2.cpp.o
[  3%] Building CXX object dlib/CMakeFiles/dlib.dir/bit_stream/bit_stream_kernel_1.cpp.o
[  4%] Building CXX object dlib/CMakeFiles/dlib.dir/entropy_decoder/entropy_decoder_kernel_1.cpp.o
[  4%] Building CXX object dlib/CMakeFiles/dlib.dir/entropy_decoder/entropy_decoder_kernel_2.cpp.o
[  5%] Building CXX object dlib/CMakeFiles/dlib.dir/entropy_encoder/entropy_encoder_kernel_1.cpp.o
[  6%] Building CXX object dlib/CMakeFiles/dlib.dir/entropy_encoder/entropy_encoder_kernel_2.cpp.o
[  7%] Building CXX object dlib/CMakeFiles/dlib.dir/md5/md5_kernel_1.cpp.o
[  8%] Building CXX object dlib/CMakeFiles/dlib.dir/tokenizer/tokenizer_kernel_1.cpp.o
[  8%] Building CXX object dlib/CMakeFiles/dlib.dir/unicode/unicode.cpp.o
[  9%] Building CXX object dlib/CMakeFiles/dlib.dir/data_io/image_dataset_metadata.cpp.o
In file included from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads.h:22:0,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/matrix_la.h:25,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix.h:11,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/point_transforms.h:9,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry.h:10,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/image_dataset_metadata.h:8,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/image_dataset_metadata.cpp:6:
/home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/thread_pool_extension.h:460:26: error: ‘exception_ptr’ in namespace ‘std’ does not name a type
             mutable std::exception_ptr eptr; // non-null if the task threw an exception
                          ^
/home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/thread_pool_extension.h: In constructor ‘dlib::thread_pool_implementation::task_state_type::task_state_type()’:
/home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/thread_pool_extension.h:419:107: error: class ‘dlib::thread_pool_implementation::task_state_type’ does not have any field named ‘eptr’
             task_state_type() : is_being_processed(false), task_id(0), next_task_id(2), arg1(0), arg2(0), eptr(nullptr) {}
                                                                                ^
/home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/thread_pool_extension.h: In member function ‘void dlib::thread_pool_implementation::task_state_type::propagate_exception() const’:
/home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/thread_pool_extension.h:464:21: error: ‘eptr’ was not declared in this scope
                 if (eptr)
                     ^
/home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/thread_pool_extension.h:466:32: error: unable to deduce ‘auto’ from ‘eptr’
                     auto tmp = eptr;
                                ^
/home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/thread_pool_extension.h:468:21: error: ‘rethrow_exception’ is not a member of ‘std’
                     std::rethrow_exception(tmp);
                     ^
In file included from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/parallel_for_extension.h:9:0,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads.h:24,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/matrix_la.h:25,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix.h:11,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/point_transforms.h:9,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry.h:10,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/image_dataset_metadata.h:8,
                 from /home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/image_dataset_metadata.cpp:6:
/home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/async.h: In lambda function:
/home/rohit/Desktop/HiSillicon/dlib/dlib/data_io/../geometry/../matrix/../threads/async.h:73:37: error: ‘current_exception’ is not a member of ‘std’
                 prom->set_exception(std::current_exception());
                                     ^
dlib/CMakeFiles/dlib.dir/build.make:326: recipe for target 'dlib/CMakeFiles/dlib.dir/data_io/image_dataset_metadata.cpp.o' failed
make[2]: *** [dlib/CMakeFiles/dlib.dir/data_io/image_dataset_metadata.cpp.o] Error 1
CMakeFiles/Makefile2:85: recipe for target 'dlib/CMakeFiles/dlib.dir/all' failed
make[1]: *** [dlib/CMakeFiles/dlib.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

My cmake command is like:
cmake -DCMAKE_C_FLAGS="-O3 -mfpu=neon -fprofile-use -DENABLE_NEON" -DNEON=ON -DCMAKE_C_COMPILER=/opt/hisi-linux/x86-arm/arm-hisiv600-linux/target/bin/arm-hisiv600-linux-gcc -DCMAKE_CXX_COMPILER=/opt/hisi-linux/x86-arm/arm-hisiv600-linux/target/bin/arm-hisiv600-linux-g++ -DCMAKE_CXX_FLAGS="-std=c++11" -build -config Release ..
The compiler itself have support for -std=c++11. Is there any work around this or any cmake flag which i can use while build ? Any help will be hugely appreciated. Thanks in advance.

Hi, I just wanted to note I have successfully cross-compiled dlib for arm a few weeks ago, so I rather doubt  it is a problem with dlib. I used a linaro toolchain on windows, though.

Thanks for your reply. It was compiler issue which is not related to dlib and after spending some time, i successfully finished compiling. Closing this now.

I have the same problem. How can I solve it？


[master] Bump Mesos to nightly master 49cab93

High-level description
This is a routine bump to the latest Mesos and mesos-modules.
Related JIRA Issues
DCOS-43593 - Some Mesos master endpoints do not handle failed authorization properly.
Checklist for all PRs

[x] Included a test which will fail if code is reverted but test is not. If there is no test please explain here:
[x] Read the DC/OS contributing guidelines
[x] Followed relevant code rules Rules for Packages and Systemd

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[x] Changelog:
https://github.com/apache/mesos/compare/22fa5aeac19416fd0c7c2284a1e48a5ee15bdd24...49cab93b565d250fd7e88cd8bb9cd49abbdd7da4

[ ] Test Results: [link to CI job test results for component]

[ ] Code Coverage (if available): [link to code coverage report]

@mesosphere-mergebot label Ship It


Fix retry_codes bug in config json

Codecov Report

Merging #60 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #60      +/-   ##
==========================================
+ Coverage   99.43%   99.43%   +<.01%
==========================================
  Files          13       13
  Lines        1421     1426       +5
==========================================
+ Hits         1413     1418       +5
  Misses          8        8

Impacted Files
Coverage Δ

lib/google/gax/api_callable.rb
98.33% <ø> (ø)
:white_check_mark:

spec/google/gax/settings_spec.rb
100% <ø> (ø)
:white_check_mark:

lib/google/gax/settings.rb
98.98% <100%> (+0.05%)
:white_check_mark:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f7c65a9...fd41fe2. Read the comment docs.

Woo! 🎉


Sentiment Annotator is not working

Hello,
I was trying to get this working on some customer related data. None of the hacks are working although through command line I can run the sentiment analysis. Please advise as I don't want to switch to Java for this. Couldn't find anything on sentiment using Stanford CoreNLP APIs.
from corenlp_pywrap import pywrap
full_annotator_list = pywrap.CoreNLP.annotator_full_list + ['sentiment']
cn = pywrap.CoreNLP(url='http://localhost:9000', annotator_list=full_annotator_list)
out = cn.basic("this movie was good but highly overrated", out_format='json')
print(out.content)
Thanks.
AB

Very much appreciated, if you could solve this. I am looking to use sentiment annotator too. Thanks
S.W.

I can confirm that it is an issue in the wrapper. I just download the most current corenlp server, compile it as jar, then run the server. I am able to get sentiment result from the server via standard method: annotator sentiment. However it is not working with sentiment annotator with python wrapper.
S.W.

Well, as a work around I wrote a code to get me the sentiment scores in Java(took the code from stanford sentimentannotator class) and used Pyjnius to wrap around the code to use in Python. Here, issue might be(I guess) the Stanford CoreNLP API is not supporting the sentiment annotator right now or I am missing something to get it working.
Thanks.
AB

@AbhinavBhatnagar Thanks for the heads up. it seems Pyjnius is not working in python 3.5. also another issues, it seems that I am not able to load StanfordCoreNLP.java properly, so I am not able to construct StanfordCoreNLP. any idea?

@AbhinavBhatnagar I know this could be a big ask. could you please share your sentiment scores java code with me? for some reason, my java code is not working. I am new to Java. I am using "Py4j" to access JVM. my email<EMAIL_ADDRESS>really appreciate it.
S.W.

ok, I also have it working via py4j,wrote a java program.

My bad @sceamms I have been writing a wrapper for you in python. Great!!

If its working manually, it should work with pywrap. Can you guys share some screeshot or the code you use for running it manually. Let me check for some subtle bugs

@AbhinavBhatnagar Thank your for your kindness. appreciate it!
@hhsecond there are 2 options depends on your need of sentiment: if you only need sentiment result based on the sentence.e.g. Positive, Neutral, Negative.
It is still to use corenlp server, but the official published release "stanford-corenlp-full-2015-12-09", there is a bug in it, sentiment annotator cannot be added to the server. It is corenlp bug. However you can download the latest corenlp (you need to compile it yourself) from github: https://github.com/stanfordnlp/CoreNLP
follow the instruction to compile it and create a Jar to replace stanford-corenlp-3.6.0.JAR in the working directory of stanford-corenlp-full-2015-12-09.
then you need to download the latest model via: http://nlp.stanford.edu/software/stanford-corenlp-models-current.jar
use this most current model to replace "stanford-corenlp-3.6.0-models.jar" in the folder.
then you restart the server from the web localhost:9000, you will be able to add "sentiment annotator":
if you want to have it python result, there is another wrapper "pycorenlp": it is working with sentiment annotator by default: code as below, very simple.
from pycorenlp import StanfordCoreNLP
nlp = StanfordCoreNLP('http://localhost:9000')
text = '9988??'
output = nlp.annotate(text, properties={'annotators': 'sentiment', 'outputFormat': 'json'})
print(output['sentences'][0]['sentiment'])
in my case, I need to get score distribution for some specific reason, which is a bit more complex, I can not use server directly (it does not support I believe). I need to have access some corenlp java class in python.
for normal purpose, above method should be enough..
good luck.

@sceamms https://bitbucket.org/crossviral/deepsentiment if by any chance you need it.

@AbhinavBhatnagar Very cool! Thank you. Certainly will use it as an alternative. Like to have both options.

@hhsecond I am running into the same issue. The sentiment annotator works well in the browser but gives "AssertionError: annotator not supported: {'sentiment'}" error running the python wrapper. I am running the latest Core NLP 3.7.0 so it should have sentiment support.
I'd really appreciate the help!

Hi @bwang12 , I am no longer maintaining this project. There is a new 3.7 python wrapper for CoreNLP. I haven't tested it but it should have all those features.
Even if you want to use corenlp_pywrap, the script is so simple that you can easily make changes to it. Would be happy to merge the PR if you are working on it.

Closing it due to inactivity


feat: Accept to send the portkey params in the create methods

Summary
Currently the implementation of the portkey SDK is to create a config and pass the model params on the create methods. This has been useful in most cases but in somecases users want to be able to set the portkey params on the create method over-riding the config created globally.
Source: From the discord conversation with Ragas team.
Solution
Accepting the portkey params on the create methods of the SDK

This has now been launched in the 1.0.0 SDK. Closing this as done.


Updates on Pocketbook List view

[x] Unify list of pocketbooks and let user view all the pocketbooks at once

Remove Owned | Collaborated button toggle on pocketbooks/list

[x] Click on card to open and have edit delete in menu - do not show edit delete upfront
[x] No FAB in desktop - Pocketbook list view header should have button to Add Pocketbook

Add a placeholder card to add a new pocketbook which would be visible at all times

WIP as of Aug 14

Show Contributors modal and string
Actions menu for a pocketbook


fix: adding .js extension to exported *.css.js file

• this was throwing off some versions/configurations of Jest
• the module loader was looking for a CSS file
Signed-off-by: Scott Mathis<EMAIL_ADDRESS>PR Checklist
Please check if your PR fulfills the following requirements:

[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)
[ ] If applicable, have a visual design approval

PR Type
What kind of change does this PR introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] clarity.design website / infrastructure changes
[ ] Other... Please describe:

What is the current behavior?

We have to export the base element styles. These are generated *.css.js styles.
Some configs/implementations of Jest don't try to look for *css.js. They stop looking at *.css.
Issue Number: N/A
What is the new behavior?
I added the full path. This is the only Clarity issue I've been able to find that impedes Jest configuration. Every other issue in #4094 and #4196 are (thus far) solvable by properly configuring an application.
Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information

I have no idea if we want to leave #4094 and #4196 open after this is merged.
No other issues in those tickets are caused by or related to Clarity.

@gnomeontherun
See #4365


Release Google.Analytics.Data.V1Alpha version 1.0.0-alpha05

Changes in this release:
No API surface changes; just dependency updates.

Created release for Google.Analytics.Data.V1Alpha-1.0.0-alpha05


how to log to blob storage as a .csv file?

I have created a .net core 2.1 web app, and enable the "Application Logging(blob)".
And in my code startup.cs -> Configure method, the following code is added:
loggerFactory.AddAzureWebAppDiagnostics(
                new Microsoft.Extensions.Logging.AzureAppServices.AzureAppServicesDiagnosticsSettings
                {
                    OutputTemplate="xxxx",
                    BlobName ="logs.csv"
                }
                );

But after published to azure , run it. Then in the blob I specified earlier, it shows the .txt log file, not .csv.
Can somebody let me know how to log it as a .csv file?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 46653d6e-d057-e698-ef94-32d48e12d39f
Version Independent ID: 74beaf3c-60f9-006d-110a-fbe2348d4797
Content: Enable diagnostics logging for web apps - Azure App Service
Content Source: articles/app-service/web-sites-enable-diagnostic-log.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

Hi @imeya, thanks for the feedback! We are currently investigating and will update you shortly.

Hi @imeya, currently .csv files are not supported. You can request this feature from the product group here.
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


Transient hostname is being set on reboot

Currently NWK1's kubic cluster upon reboot is losing it's persistence for a static hostname and going to a Transient hostname.
Add automation upon provisioning to correct this situation.

This is now resolved by setting kernel level sysctl.

This is now resolved by setting kernel level sysctl.


Error!

I'm not sure what is it, so my description  of the issue is poor.
But here is the screenshot and I hope it helps! :-)

Hey @chraibi
Thank you so much for reporting this error. Due to time constraints in the hackathon, I did not fully test all the features of the app and realized all the PNG files were resulting in this error because I missed one important step before passing the image to the model.
It has been fixed now. Do try again and reply to this message if it still doesn't work.
Thanks,
Kanak

works!


Document Prometheus configuration with Antrea

Document the interoperability of Antrea controller and agents with
Prometheus monitoring server.
Add a document describing configuration changes in Antrea and Prometheus.
Also add a sample yaml file.

/skip-all


[Bug]: generateDataTreeWidget is slow performing causing the app ui navigation to be slow

Is there an existing issue for this?

[X] I have searched the existing issues

Description
generateDataTreeWidget function is causing slow apps. Relevant slack thread here
Steps To Reproduce
NA
Public Sample App
No response
Environment
Production
Issue video log
No response
Version
Cloud

Duplicate of #31267


Creating action creators in Flow, could actions that are classes be supported?

I've been playing around with react redux and flow in some projects. I like it, but there is one thing I hate: adding an action creator. The problem is, that I have to change stuff, in 3 files, and it is very repetitive. I will show an example:
src/ActionModal/actions.js
import ActionTypes from "./ActionTypes";

export type ShowActionModal = {
  type: 'ShowActionModal'
}

function show(): ShowActionModal {
  return {
    type: ActionTypes.show
  }
}

export type HideActionModal = {
  type: 'HideActionModal'
}

function hide(): HideActionModal {
  return {
    type: ActionTypes.hide
  }
}

export default {show, hide};

src/ActionModal/ActionTypes.js
const show = 'ActionModal/show';
const hide = 'ActionModal/hide';

export default {show, hide};

src/Action.js
import {Increment, Decrement} from './Counter/actions';
import {ShowActionModal, HideActionModal} from './ActionModal/actions';

export type Action = Increment | Decrement | ShowActionModal | HideActionModal;

I have to type 4 times the word ShowActionModal, just to add one simple action creator to my project. This just seems like so much work, for just two simple actions.
I'm wondering if you guys have thought about supporting the following pattern:
src/ActionModal/actions.js
class ShowActionModal extends Action {}
class HideActionModal extends Action {}

src/Action.js
class Action {
  constructor() {
    // here you can for example log that an action is created ...
    // or nothing :-)
  }
}

This is much less code. There is no need for hardcoding an action constant. And now you can just change one file, anytime you create an action creator (the constructor is now the action creator). There is no need to declare a type, because a class is a type.
The reducer would look like this:
function showModalReducer(state, action) {
  switch (action.constructor) {
    case ShowActionModal:
      return true;
    case HideActionModal:
      return false;
    default:
      return state;
  }
}

Also if you would have some more complex actions, you can do this like this of course:
class ToggleTodo extends Action {
  id: number;
  constructor(id: number){ this.id = id; }
}

Tagging @gcanti as I saw that he has some experience with using redux with flow.

This is not a scenario that will be directly supported in Redux as far as I know.  There are, however, a number of community-built libs that implement more OOP-style approaches like this.  See https://github.com/markerikson/redux-ecosystem-links/blob/master/variations.md for links to some of them.

@markerikson Okay, I see, but I hoped that this could be considered as this would help a lot with reducing boilerplate in the flow case, that I heard @gaearon wants to have good support for.
I just found out that I could implement this myself in this way:

Don't use createStore from redux library
Copy the createStore file from redux, and remove the following lines:

    if (!isPlainObject(action)) {
      throw new Error(
        'Actions must be plain objects. ' +
        'Use custom middleware for async actions.'
      )
    }

    if (typeof action.type === 'undefined') {
      throw new Error(
        'Actions may not have an undefined "type" property. ' +
        'Have you misspelled a constant?'
      )
    }

Rewrite the object { type: ActionTypes.INIT } as a class
Use this new createStore function in your redux app

I can't speak to the Flow aspect, but I know Dan has said a number of times that "reducing boilerplate" is not an objective for Redux, and that if you really want to you can build your own abstractions.  See https://twitter.com/dan_abramov/status/733742952657342464, http://redux.js.org/docs/recipes/ReducingBoilerplate.html , and https://github.com/reactjs/redux/issues/1171#issuecomment-196819727 for some thoughts on that topic.

Dan has said a number of times that "reducing boilerplate" is not an objective for Redux

I understand and appreciate that way of thinking. However, have you read my example. Maybe I should make my example shorter, to make the amount of boilerplate more clear in flow:
// file 1
import ActionTypes from "./ActionTypes";

export type ShowActionModal = {
  type: 'ShowActionModal'
}

function showActionModal(): ShowActionModal {
  return {
    type: ActionTypes.ShowActionModal
  }
}

// file 2
const ShowActionModal = 'ShowActionModal';

// file 3
import type {ShowActionModal} from './ActionModal/actions';
export type Action = ... | ShowActionModal

actually, if you think about it like this, this is 9 time writing the same word, in 3 different files, to finally get an action creator working. I think the flow case is quite much worse, than the normal case.
So if you would support action to be a class (maybe only support classes without methods, so that they are serializable?), you can reduce all this boilerplate, to the following line:
import '../Action';

class ShowActionModal extends Action{}

I will read a bit more about the links you send me about the concerns about classes. I'm wondering, is this all about, that classes are not serialisable? Is that the concern? Because I don't want a class because I want to add fancy methods to my action, I just want my Action to have only serializable properties. The only reason I like the class, is because it saves me from writing 9 time the same word, in 3 different files.

Yeah, that's the biggest factor.  The biggest reason for Redux's existence is to be able to serialize state and actions to enable time-travel debugging.  The vast majority of how it's used falls out from that (plain objectx for actions, with type fields that use strings as constants, etc).  Class instances are not serializable, and doing instanceof checks gets really hairy.
Like I said, you're certainly welcome to come up with your own abstractions on top of Redux, but the core is not likely to be changed much at this point, and definitely not for something like this (see #1813  for an example of this type of discussion and Dan's thinking).  There are other libs out there that put an OOP-type abstraction on top of actions and reducers, in the list I linked.  MobX is also a lot more object-oriented, and generally regarded to have less boilerplate involved.
Ultimately, the main goals for Redux are things like predictable / functional-style state updates, and enabling scenarios like time-travel debugging.  Aspects such as serializability and some of the "boilerplate" do generally fall out as results of those goals.

@markerikson

The biggest reason for Redux's existence is to be able to serialize state and actions to enable time-travel debugging. The vast majority of how it's used falls out from that (plain objectx for actions, with type fields that use strings as constants, etc). Class instances are not serializable, and doing instanceof checks gets really hairy.

Okay, well, fair enough, I'm not really convinced that using classes that behave like plain objects, would make time travelling impossible. But that is probably my lack of understanding.
Thanks for the links to MobX etc. but I like redux a lot. I like the time travelling, I just want to get flow support without feeling like writing redundant code all the time.
I actually found a way to do this, while still dispatching plain objects.
Using this class:
export default class Action {
  type: string;

  constructor() {
    this.type = this.constructor.name;
  }
}

Then override the dispatch method of the store
let oldDispatch = store.dispatch;
store.dispatch = (action) => {
  if (action instanceof Action) {
    action = {...action}
  }
  oldDispatch(action);
};

Then write actions like this:
class Increment extends Action {}
class Decrement extends Action {}

And reducers like this:
export default function reducer(counter: number = 0, action: Action): number {
  switch (action.type) {
    case Actions.Increment.name:
      return counter + 1;
    case Actions.Decrement.name:
      return counter - 1;
    default:
      return counter;
  }
}

What about a middleware that turns a class-based action into a plain object action where the type property is the class name? Then you could define your actions as classes, but redux core would be satisfied.

@jimbolla Thanks for the tip. Yes, this actually works pretty good.
I do it now like this, in the middleware, I convert the action class object, to an plain object.
And I wrap the rootReducer into a function that wraps the plain action objects back to to the class objects.
Works fine :)
export const store = createStore(
  convertPlainObjectToAction(rootReducer, Actions),
  {},
  compose(
    applyMiddleware(convertActionToPlainObject(Action)),
    window.devToolsExtension ? window.devToolsExtension() : f => f
  )
);

function convertPlainObjectToAction(reducer: Reducer<*, *>, Actions: Object): Reducer<*, *> {
  return (state, action) => {
    if (Actions.hasOwnProperty(action.type)) {
      const classAction = new (Actions[action.type])();
      return reducer(state, classAction);
    }
    return reducer(state, action);
  }
}

function convertActionToPlainObject(Action: Class<Action>) {
  return () => {
    return (next) => (action) => {
      if (action instanceof Action) {
        action = {...action}
      }
      return next(action);
    }
  }
}

@kasperpeulen with flow in place you may want to discard constants (replaced by flow type checking), so no ActionTypes file.
IMO actions creators should be defined only when strictly necessary (for example precisely in order to reduce the boilerplate) not something that you always do.
However flow introduces some boilerplate in its own, a price that I'm glad to pay if you ask me.
Note how you are losing type safety here
class Action {
  type: string;

  constructor() {
    this.type = this.constructor.name
  }
}

class Increment extends Action {
  step: number
}

export function reducer(counter: number = 0, action: Action): number {
  switch (action.type) {
    case Increment.name:
      return counter + action.step; // <= property `step`. Property not found in Action
    case Decrement.name:
      return counter - 1
    default:
      return counter
  }
}

@gcanti

with flow in place you may want to discard constants (replaced by flow type checking), so no ActionTypes file.

Yes, you have a point about the action types. The downside is that you miss the speed and explorability feature of auto completion.

IMO actions creators should be defined only when strictly necessary (for example precisely in order to reduce the boilerplate) not something that you always do.

Yes, I have been wondering about that too. I just create action creators, because people say that this is the way to go. But you mean just dispatching the object itself instead?

Note how you are losing type safety here

Yes, I just noted. I have no other solution that I think is kind of neat:
store.js
import {compose, createStore, applyMiddleware} from 'redux';
import rootReducer from './reducer';
import Action from './Action';
import Actions from './Actions';

const convertPlainObjectToAction = createStore => reducer => createStore(
  (state, action) =>
    Actions.hasOwnProperty(action.type)
      ? reducer(state, new (Actions[action.type])())
      : reducer(state, action));

const convertActionToPlainObject = () => next => action =>
  (action instanceof Action)
    ? next({...action})
    : next(action);

export const store = createStore(
  rootReducer,
  {},
  compose(
    convertPlainObjectToAction,
    applyMiddleware(convertActionToPlainObject),
    window.devToolsExtension ? window.devToolsExtension() : f => f
  )
);

Action.js
// @flow
export default class Action {
  type: string;

  constructor() {
    this.type = this.constructor.name;
  }
}

Actions.js
// @flow
import Action from '../action';

class Increment extends Action {}
class Decrement extends Action {}

export {Increment, Decrement};

Dispatch like this: dispatch(new Actions.Increment())
And the reducer is like this:
export default function reducer(counter: number = 0, action: Action): number {
  if (action instanceof Actions.Increment) {
    return counter + 1;
  } else if (action instanceof Actions.Decrement) {
    return counter - 1;
  } else {
    return counter;
  }
}


No progress bar when running subsequent ServiceX queries

This is probably a ServiceX issue rather than a coffea-casa one, but if you run two ServiceX queries, e.g.:
jet_p4 = ds.Where(lambda event: event.nJet > 0)\
                    .Select(lambda event: Zip({'pt': event.Jet_pt,
                                        'eta': event.Jet_eta,
                                        'phi': event.Jet_phi,
                                        'mass': event.Jet_mass}).ToFourMomenta().Where(lambda jet: jet.pt > 30)).AsAwkwardArray().value()

muon_p4 = ds.Select(lambda event: Zip({'p4': Zip({'pt': event.Muon_pt,
                                                  'eta': event.Muon_eta,
                                                  'phi': event.Muon_phi,
                                                  'mass': event.Muon_mass}).ToFourMomenta(),
                                           'isMed': event.Muon_mediumId
                                     })
                       .Where(lambda light_mu: light_mu.isMed == 0)
                       .Where(lambda mu_pt: mu_pt.p4.pt > 10)
                       .Select(lambda mu: mu.p4)).AsAwkwardArray().value()

You'll get a progress bar for the first (in this case, jet_p4) but not for the second (in this case, muon_p4). This makes it look like the JupyterLab has frozen until the second query is also done processing.

Thanks a lot @mat-adamec for user experience feedback! I will ping @BenGalewsky to help us understand if it is something on ServiceX side and we need to move this issue to Servicex git repo or it is on coffea-casa side?

Yes, moved to ServiceX_Frontend Issue 218


Add status to node table

Fixes #629
This is how it looks like when a node drain:

"Leaving" does not has a meaning in Scylla
we have in nodetool status for example
UN - Up Normal
UJ - Up Joining
UL - Up Leaving (decomissioning)
Is decomission and drain showing up the same it shouldn't we should have Drained

it shows what the node think about its status if it's a value a node can return, it should be mapped to something

Well its not leaving
Drained nodes should show up as Down AFAIK

The description is taken from the enum mapped to the metric:
https://github.com/scylladb/scylla/blob/91b71a0b1a5a18b520327e87df52b88b7a715e09/service/storage_service.cc#L256


Add SuperLU_MT support

It would be nice to have the option of using SuperLUMT with the sparse matrix support, since KLU is for quite weird circumstances. We'd first have to update the artifact to build with it, then add a branch to our linear solver support for it.

@jd-lara Do you know if SuperLU is the only supported direct sparse solver for Sundials. If we could use UMFPACK, it may be trivial to enable.

KLU and SuperLU are the only two that are supported, other than things like SuperLUDist which require a different NVector type.

I had a conversation with @balos1 a few months ago about the implementation of custom linear solvers with the intention of implementing pardiso. It is not too bad to write the Sundials Wrapper for custom linear solvers.

We could use that so that way it works with the DiffEq linear solver support, and that would then allow UMFPACK.
Ref: https://docs.sciml.ai/latest/features/linear_nonlinear/
@YingboMa is currently improving this part of DiffEq a lot, so it might be good to have him in the loop here.

We already have MKL Pardiso in BB. I am not sure if SuperLU MT receives bugfixes. How important is this feature to enable in Sundials?

@ChrisRackauckas Are you suggesting that have Sundials call out to the julia linsolve, and then use that to pull in any sparse direct solver?

Yes. Otherwise SuperLU is the only option left in the list that Sundials allows. So if we plan on linking in more, we can just put it to the SciML linsolve interface, and then any Julia code works there and we can call it done.

looks like there is already a wrapper for SuperLUMT
https://github.com/LLNL/sundials/tree/master/src/sunlinsol/superlumt

Yes. Otherwise SuperLU is the only option left in the list that Sundials allows. So if we plan on linking in more, we can just put it to the SciML linsolve interface, and then any Julia code works there and we can call it done.

In order to achieve this, these methods need to be implemented
https://github.com/LLNL/sundials/blob/master/src/sunlinsol/klu/fsunlinsol_klu.h

Yes.

This is issue is that, of all of the ones that don't require a new special array type, SuperLU MT is the last one in the list to do. So we probably should do it, or just add our SciML linear operator interface to this package, in which case we'd get UMFPACK support for free from Julia Base, + MKLPardisio + etc.

Adding SuperLU MT is much lesser effort, I suspect. Do we know if it links to 64-bit BLAS and if that combination works with Sundials? If not, no problem, because we now also have 32-bit BLAS in BB.

Adding SuperLU MT is much lesser effort, I suspect. Do we know if it links to 64-bit BLAS and if that combination works with Sundials? If not, no problem, because we now also have 32-bit BLAS in BB.

From the documentation, it seems that it depends on how SUPERLUMT is built.
At the same time, the variable SUPERLUMT LIBRARIES must be set to a semi-colon separated list of other libraries SuperLU MT depends on. For example, if SuperLU MT was build with an external blas library, then include the full path to the blas library in this list. Additionally, the variable SUPERLUMT THREAD TYPE must be set to either Pthread or OpenMP.

@christopher-dg It seems you might be looking into this. Just checking, so that we don't end up doing the same things. It would be great to have more help maintaining this whole ecosystem of scientific software.

Yeah I was looking into building a SuperLU_MT_jll to then be used here.

Any updates on using Julia linear solvers within sundials?
I'm also trying to figure out whether there's code anywhere that gets Pardiso to work with sundials.

Aren't you better off just using Julia's DiffEq at that point?

Pardiso isn't compatible with the the Sundials C code at the moment. Currently we the only linear solver not supported is SuperLUMT.

Yes, Sundials has a pretty strict set of linear solvers. If you want to use Pardiso you can use it with OrdinaryDiffEq as documented on https://diffeq.sciml.ai/stable/features/linear_nonlinear/#Pre-Built-Linear-Solver-Choices . Given the performance and generality advantages of OrdinaryDiffEq, it makes a lot more sense to just do that anyways.

I will take another swing at this during the summer to improve benchmarking against some of the DiffEq solvers for stiff systems. But for now, the addition of interfaces for more linear solvers is very time-consuming and it seems that there isn't an immediate need to support more linear solvers.

I tried solvers from DiffEq, the costliest steps for my use case were LU factorizations followed by function/Jacobian calls, both of which CVODE seemed to reduce the number of, compared to the DiffEq solvers.
so it seems the best option for me is to use a performant sparse solver within CVODE (if possible a Cholesky solver since my jacobian is symmetric too).

I tried solvers from DiffEq, the costliest steps for my use case were LU factorizations followed by function/Jacobian calls, both of which CVODE seemed to reduce the number of, compared to the DiffEq solvers.

With TRBDF2 and KenCarp47? Can you share the example?

so it seems the best option for me is to use a performant sparse solver within CVODE (if possible a Cholesky solver since my jacobian is symmetric too).

LinSolveFactorize(cholesky!) should do it for the pure Julia solvers.

BTW, the latest release post shows QNDF outperforming CVODE_BDF across the board, which renders this issue somewhat moot: https://sciml.ai/news/2021/05/24/QNDF/

Thanks! will check it out. (was gonna write up an example independent of the libraries we use next weekend)


Would close DNN-8479

This would allow the installer to include pdb files in the installation, as described in detail in https://dnntracker.atlassian.net/browse/DNN-8479
Here an excerpt:
When users install our modules and run into an error, they have trouble reporting what exactly the error is, because the stack in the events-module is not really helpful.
BUT there is a solution: if we deploy the built PDB files from visual studio together with the .DLL files, then the error messages include exact line numbers. This is very helpful in fixing issues.
PDB files have no security implication, they are like JS-map files and only help the error-system inform the user about the code-line where something happened.
Because of this, it's important that we can also include the PDB files in the installation. Unfortunately as of now, the installer blocks this extension - telling the user it's unsafe.
We should add pdb-files to the list of allowed files in the assembly installer to aid in better user error-feedback.

The symbols packages that DNN generates don't show any errors (they use a ResourceFile component to place the pdb files in the bin folder).  Does that work for you, or does using the Assembly component get you some other benefit?

For the record, I have no problem with this change, just wanting to better understand the different options.

I'm now also using the resourcefile workaround. It works, but it's very hacky / dirty / indiscoverable. IMHO this should be straight-forward and not feel like you're abusing a back-door.

I believe the ResourceFile workaround is only valid for only a single package with pdb's (for uninstallation that is). It would be great to support pdb files along side their dll components. Thanks for the contribution @iJungleboy!

Will drop this pull - and re-create as it became stale...

Note: this looks long, but it's only 1 simple file that changes!

Will cancel this and send a new pull-request with just this micro-change...
https://github.com/dnnsoftware/Dnn.Platform/pull/1769

Second micro-change in this new pull request: https://github.com/dnnsoftware/Dnn.Platform/pull/1770


Bug with Range Validator :: min and max as functions are not handled properly

With RangeValidator one can use functions in place of values for min and max options. Like this:
  validates "foo", range:
    min: -> 0.5 * @get "bar"
    max: -> 2 * @get "bar"

When you run validation first time on it, it will replace @options.min and @options.max with respective values returned from functions. And after that these functions will never be called again. Even on different instances of model.
This PR fixes it by copying min and max functions to @options.minFunc and @options.maxFunc when they are functions. And uses these options to call on target object and writes return value to @options.min and @options.max respectively.

:+1: thanks!


Make the Discover page mobile responsive

We want the Discover page, which is the homepage of our web app mobile responsive.  Please reach out to me if you have questions
TODO

[ ] This page should be responsive on all mobile devices
[ ] This page should be responsive on regular desktop screens
[ ] This page should also be responsive and display properly on very large screens i.e monitors
[ ] Please rely on tailwind's responsive classes as much as possible

N:B - This does not include making the Player responsive

Working on this issue, please assign it to me