Minor UI issue while enabling the project monitoring

Internal reference: SURE-4734
Reported in 2.6.5
Issue description:
If you select a project in the dropdown list, you can set up your installation. But if you click on the selected project a second time, the UI doesn't show up anything anymore.
Repro steps:

Enable the project monitoring by installing the corresponding charts
Navigate to Cluster-Monitoring-Project Monitors
Click on create (https://$IP_ADDRESS/dashboard/c/c-2q8b6/monitoring/helm.cattle.io.projecthelmchart/create)
Select the project, it will display AlertManager, Grafana,federation tabs
Again select the same Project

Workaround:
Is workaround available and implemented? yes
What is the workaround: select other project again click on same project.
Actual behavior:
it is not showing AlertManager, Grafana,federation tabs if we select the same project again. Able to reproduce the issue in my environment. Not sure is it intentionally implemented in this way or its an UI  bug.
Expected behavior:
User expects it should show same UI components if we select same project again.

Possibly tied to SURE-4692 as well.
Issue description:
with the Prometheus federator installed, the project owner cannot install project monitoring…they get this error "Method POST Not Supported"
Repro steps:

Install the Prometheus federator
try to enable the project monitor with account having project owner role.

Workaround:
Is workaround available and implemented? yes
What is the workaround: able to enable with admin account.
Actual behavior:
It is showing "Method POST Not Supported" when we try to enable the Project monitoring with project owner role.
Expected behavior:
it should enable project monitoring  for an account having project owner roles, however if the account(user) not have proper permission, then it should show a proper error messages saying "permission not enough to enable the project monitoring".
Additional notes:
Via someone's repro in 2.6.5, it seems the issue is coming before installing the ProjectHelmChart/ project monitor operator. After installing ProjectHelmChart, they're not not able to reproduce; it is pushing the CRD,RBAC role binding properly.

Additional update from SURE-4692:

Users still cannot enable project monitoring....when they try to install the helm_project_operator they receive the following:
Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: could not get information about the resource PodSecurityPolicy "helm-project-operator-psp" in namespace "": podsecuritypolicies.policy "helm-project-operator-psp" is forbidden: User "u-q6str" cannot get resource "podsecuritypolicies" in API group "policy" at the cluster scope

I don't believe 4692 is related (that one seems more RBAC related than anything) but I don't seem to be able to reproduce this issue as described. My steps are as follows:

Create new cluster for Project Monitoring Test (RKE2) using Rancher 2.6.6
Install Monitoring Chart
Install Prometheus Federator (uncheck "Enable Embedded Helm Controller" as this is an RKE2 cluster)
Create "projectA" project and "namespace-a" namespace, notice "cattle-project-" namespace is also created
Create "projectB" project and "namespace-b" namespace, notice "cattle-project-" namespace is also created
Navigate to "Monitoring > Project Monitor"
Click "Create" button in top-right corner
Select "projectA" in "Project" dropdown
Reopen "Project" dropdown and select "projectA" again
Noticed that form tabs ("Alertmanager", "Grafana", "Federation") are still present.

Also tried:
Reproducing when "projectA" already has Project Monitoring installed
Reproducing using "projectB" when "projectA" already exists
Reproducing with "default" project

Jumped on a call with Suresh to reproduce. We were unable to reproduce in 2.6.7 but we were able to reproduce in 2.6.5. It looks like the fix for this was a side-effect of another fix in 2.6.7 and should be resolved by upgrading to 2.6.7 on release. Putting into "Test" to independently verify that the issue is resolved in the upcoming release.

retest: Passed
retest cluster information
**rancher **- 2.6.7-rc5
rke2 version- v1.24.2+rke2r1
I followed @Sean-McQ repro steps as well as some adhock testing, I can confirm that this bug is no longer reproducible. closing ticket.

This is still being reported as an issue

SURE-4692 has become active again, so putting this back on our radar for triage.

@catherineluse once you've tackled over your other priority tickets, see what you can do here based on past efforts.

Update from @victorcasado based on one impacted user based on questions from @Sean-McQ:

Which version of monitoring and project-federator are installed?

Monitoring is v100.1.3+up19.0.3 and Project Federator is prometheus-federator:0.1.1

Verify rancher version in the dashboard (as mentioned, it’s 2.6.8 but just verifying amidst all the prior screenshots from 2.6.5)

2.6.8 confirmed

Upon the problem occurring, bring up the console and check for console errors

where to look exactly? which pod?

As well, check the network tab when reproducing and look for red rows, capturing any errors.

based on HAR file (which needs scrubbing), only one red line:  403 spotted when trying to access /k8s/clusters/c-8kc8w/v1/helm.cattle.io.projecthelmcharts

Does this work for users with different permissions? It’s mentioned in a comment that a project member can’t but can project owners and admins enable without issue?

project owners CAN enable monitoring (this is NEW)

Can you get me a list of namespaces in the projects that reproduce these errors? Is this all namespaces or some?

all

The console in question here is in the web-browser's developer tools. In chrome, you can follow these steps to open the console and get the information.
The first thing that jumps out at me is the 403 error you're seeing on /k8s/clusters/c-8kc8w/v1/helm.cattle.io.projecthelmcharts. A 403 (which refers to a "Forbidden" response) means that the user is authenticated properly but lacks the permissions to access the resource in question. Basically, this indicates to me that this is a permissions issue, if we can get somebody from the backend to clarify which permissions might be missing to cause a 403 on this resource, we can check on the frontend and then disable the action if the permission isn't present.

Update from @MKlimuszka:

https://github.com/rancher/prometheus-federator/blob/main/charts/prometheus-federator/0.1.1/charts/helmProjectOperator/templates/clusterrole.yaml
These are the permissions that we use by default. If the UI has more questions about this, let [the backend at Team 3] know.
If the reporter wants a member to be able to have more permissions, they need to assign a role binding attached to one of those cluster roles within the project registration namespace.
Also, based on one of the recent comments on this ticket, project owners can enable project monitoring, so it sounds like the original issue reported in this ticket is fixed and if there is a new bug, a new ticket should be opened.

I need to reread this and see how https://github.com/rancher/dashboard/pull/7294 maps to addressing the situation.


Nullable float support in details view

Great! I finally can define primary key other than id (blame apple, coredata), thanx!
But I have a nullable float column, and when try to view details, I get this error message
builtins.TypeError
TypeError: type NoneType doesn't define __round__ method

Is there any easy fix?

Try now

nope. made a pull request #2 with a similar fix — it was easy to copy-paste your change to one more place


.predict_proba_one(x) outputs 0.0 or 1.0 only

Hello
It seems like the predict_proba_one(x) function outputs 0.0 or 1.0 only. I expected something like 0.231 / 0.769 for example.
I'm just iterating over a test dataset with a trained LogisticRegression model (85% acc):
for x, y in X_y_test:
    y_hat = model.predict_one(x)
    # Proba ?
    print(f"Game: {x['gameId']} - Frame #{x['frame']} - Target: {y} - Predict: {model.predict_proba_one(x)}")

    model.fit_one(x, y)
    metric.update(y_true=y, y_pred=y_hat)

I printed the raw dot funtion output in the predict_proba_one() function declared in glm.py and I got :
Raw dot: -192297.15713266327
Raw dot: -192297.15779714053
Game:<PHONE_NUMBER> - Frame #10 - Target: 1 - Predict: {False: 1.0, True: 0.0}
Raw dot: -620884.1535309438
Raw dot: -620884.154215456
Game:<PHONE_NUMBER> - Frame #12 - Target: 1 - Predict: {False: 1.0, True: 0.0}
Raw dot:<PHONE_NUMBER>690841
Raw dot:<PHONE_NUMBER>415837
Game:<PHONE_NUMBER> - Frame #14 - Target: 1 - Predict: {False: 0.0, True: 1.0}
Raw dot:<PHONE_NUMBER>140164
Raw dot:<PHONE_NUMBER>863821
Game:<PHONE_NUMBER> - Frame #16 - Target: 1 - Predict: {False: 0.0, True: 1.0}
Raw dot: 1205742.3135586819
Raw dot: 1205742.3138309137
Game:<PHONE_NUMBER> - Frame #18 - Target: 1 - Predict: {False: 0.0, True: 1.0}
Raw dot: 2242763.38610308
Raw dot: 2242763.386375178
Game:<PHONE_NUMBER> - Frame #20 - Target: 1 - Predict: {False: 0.0, True: 1.0}

I feel like the values are pretty big. My model should not be that confident.
Is it normal behaviour?
Thanks

Hey there. Alas I can't give you an answer without taking a look at the data.

Hey there. Alas I can't give you an answer without taking a look at the data.

Here is the dataset I'm using : https://www.kaggle.com/bobbyscience/league-of-legends-soloq-ranked-games

I just took a quick look at your model:
model = compose.Discard('gameId', 'frame')
model |= compose.Discard('isFirstTower', 'isFirstBlood') + preprocessing.StandardScaler()
model |= linear_model.LogisticRegression(opt)

Are you sure it's doing what you want? The second line means that you're making two copies of your features, one without isFirstTower and isFirstBlood, and one where all the features scaled. Maybe what you rather want to do is:
model = compose.Discard('gameId', 'frame')
model |= compose.Discard('isFirstTower', 'isFirstBlood')
model |= preprocessing.StandardScaler()
model |= linear_model.LogisticRegression(opt)

If not I'll take a deeper look :)

@MaxHalford WOW nice catch, problem resolved I'm getting the proba! 😮
Well, I'm still discovering the pipeline composing stuff. The way I did that is because I wanted to Scale the numerical attributes and to One-Hot encode the categorical attributes.
I know how to do that with sklearn using the ColumnTransformer pipeline but not with Creme 😢
Thanks!

Actually that's something we were talking about with the team @creme-ml/developers at some point... I believe that a good idea would be to implement a SelectType transformer. This would allow to select features based on their type. It would look like this:
from creme import base

class SelectType(base.Tranformer):

    def __init__(self, dtype):
        self.dtype = dtype

    def transform_one(self, x):
        return {i: xi if isinstance(xi, self.dtype)}

This would allow doing the following:
import numbers

from creme import linear_model
from creme import preprocessing

num = SelectType(numbers.Number) | preprocessing.StandardScaler()
cat = SelectType(str) | preprocessing.OneHotEncoder()
model = (num + cat) | linear_model.LogisticRegression()

Tell me if I've missed something. We can discuss specific details. It's always great to be able to add features that make a user's life easier.

That would be a great feature! I think it will be easier if we can just pass a list of attributes just like for .Discard() and .Select() so we do not depend on the type. 😄
Thanks a lot!

I think it will be easier if we can just pass a list of attributes just like for .Discard() and .Select() so we do not depend on the type.

Can you elaborate on this?

Let's say we have extracted numerical and categorial feature column names.
num = ['numTowers', 'numKills', 'numDeaths', 'numDragons']
cat   = ['isFirstBlood', 'isFirstTower', 'gameType']

Now the user wants to normalize numerical attributes and one-hot encode the categorical ones.
num = SelectFilter(num_attrs) | preprocessing.Normalize()
cat   = SelectFilter(cat_attrs) | preprocessing.OneHotEncoder()

model = (num + cat) | linear_model.LogisticRegression()

Or simply :
num = SelectFilter('numKills', 'numDeaths') | preprocessing.Normalize()
cat   = SelectFilter('isFirstBlood') | preprocessing.OneHotEncoder()

model = (num + cat) | linear_model.LogisticRegression()

But isn't that what compose.Select already does?

You're right XD After writing it, I was like, I saw this somewhere 😭
Well, I really need to improve my pipeline skills then, I think ive missed the point.

No worries :)

I close the issue because it's resolved for me.


If actual rate equals lending rate bot should proceed with loan?

So as you can see the actual rate is same as rate set for lending, but it won't lend at that value, i guess only one if block has to be changed?
2018-12-06 11:59:44 Not lending DOGE due to rate below 0.0020% (actual: 0.0020%)

The check is
below_min = Decimal(orders['rates'][i]) < Decimal(cur_min_daily_rate)
https://github.com/BitBotFactory/MikaLendingBot/blob/c13e6b376de847060e97a86b453f9f9fca67f736/modules/Lending.py#L452
Not sure but this could be a display issue, rounding up in the log message.
`


Add code of conduct

Hey @DevTrackClub assign this issue to me i will add code of conduct, and also add Hacktoberfest tag

please assign this issue to me and add hacktoberfest tag.

Hey @DevTrackClub assign this issue to me i will add code of conduct, and also add Hacktoberfest tag

@psb27 you are assigned

Can you assign me with this issue? I think within a day I can provide the repository with a quality code of conduct. For the betterness of this project/repository.


feat:AKS Proof Claim 1.(i) Part 1

Prove Part 1 of Claim 1.(i) of Theorem 6.1 in https://www3.nd.edu/~andyp/notes/AKS.pdf.

Please remove the awaiting-author label once you have dealt with the suggestions.


Custom Mappings Not Used for Optional Properties

Using swagger-custom-mappings.yml seems to fail when properties are Option[A] wrapped types.
Our application makes heavy use of case classes that wrap strings for type safety. eg:
final case class PostalCode(value: String)
object PostalCode {
  implicit val format: Format[PostalCode] = Format(Reads.of[String].map(PostalCode(_)), Writes[PostalCode](code => JsString(code.value)))
}

These values are overridden using the scala-custom-mappings.yml file, and work as expected when they're required values. However, when the values are inside of an Option wrapper, the overrides aren't used. For example, with:
---
- type: api\.v1\.PostalCode
  specAsParameter:
    - type: string

- type: api\.v1\.Province
  specAsParameter:
    - type: string

I get the following result in the swagger.json
"api.v1.Address":{
  "properties": {
    "city":{"type":"string"},
    "province":{"type":"string"},  # Non-optional wrapper type, just like PostalCode. Works as expected.
    "postal":{"$ref":"#/definitions/api.v1.PostalCode"} # Optional, overridden type not used
  }
}

@pettyjamesm thanks for reporting the issue. To be honest I am not sure if we have time to work on this bug, but if anyone is interested, I think this shouldn't be tricky to fix. PR will be most welcome!


implemented conditional disable for export button + hover over tooltip

Also, what about the date range filter being clearable? Look at the last review main comment please @aymka .

now pleaseee tell me that nothing is missing anymore


Grouping and collapsing rows

Is there a way to group and collapse rows similar to how you can collapse columns?
In the same vein, is there any way to support 'label rows'? Similar to this example: https://www.w3.org/WAI/tutorials/tables/multi-level/#table-with-three-headers-related-to-each-data-cell. I don't see any span option, so maybe just outside of the scope of this library.

row grouping with header spans are on my todo list, i will bump them and use this issue for tracking.

Current working theory:
Add span?: readonly [number, number] to BaseGridCell. This would become the primary mechanism by which spans are defined. getCellContent to handle spans must return the same cell (by value if not by reference) for every invocation to any potential contained index inside the span.
Rendering inside drawCells will draw the spanned cell by de-clipping the original region and the re-clipping to the parent. The draw will happen on the first encounter of the span. There will be some mildly annoying math to find the true drawX location but whatever.
Handling keyboard interactions is more interesting. GridSelections which include a span do not yet have a clear indication of what should happen. I am not yet clear what the right move here is.
Open Questions
What should happen when you click on a span?
For the rest of this let us assume there are 20 columns, 5 can be visible at once, and the span goes for col 1 to 18. If you click on a span in col 4, my initial inclination is that the resulting GridSelection should be
{
  cell: [4, 10],
  range: { x: 1, y: 10, width: 18, height: 1 }
}

However this would violate one of the invariants we have inside the selection code today.
GridSelections tend to have 2 invariants:

The selection will not flow outside the actual bounds of the data
The cell will be at one of the corners of the range.

The second invariant being violated makes it unclear how to handle shift + arrow keys. Assuming the above GridSelection, Shift + Left Arrow could result in either
{
  cell: [4, 10],
  range: { x: 0, y: 10, width: 19, height: 1 }
}

or
{
  cell: [4, 10],
  range: { x: 1, y: 10, width: 17, height: 1 }
}

Both would be equally valid. Worse it seems obvious that pressing Shift + Right Arrow immediately after should essentially return to the prior state, however this would not happen regardless of which selection mode is chosen.
How to handle cells which should render centered in the visual space instead of the virtual space
A row group header should ensure the header name is always centered regardless of horizontal scrolling. Most other cell types however should not do such a thing.

You can track progress here, for now I just put in a token commit so I could open the PR and make sure I had a checklist.
https://github.com/glideapps/glide-data-grid/pull/121

Awesome, thank you! Will be following progress in the PR for sure.

I have a basic version of this. Im still not happy with keyboard interactions and selection behavior. One of the problems with spans is that range selection requires a recursive search for spans and expansion phase that basically looks like this
while selectionContainsPartialSpan() {
  expandSelectionToContainAllSpans();
}

Every time you expand the selection you may end up including more partial spans. Each check pass requires an interrogative check over all cells inside the selection (well really only new ones but who is counting?). This could become quite slow with slower data sources.

It's merged into 3.4.0

Awesome, thank you!

Do you have an example of implementing row grouping?


type dosen't infered correctly in loop when strictNullChecks turns on

TypeScript Version:  3.4.0-dev.20190310

Search Terms: loop, for, while, strictNullChecks, implicitly has type 'any' because it does not have a type annotation and is referenced directly or indirectly in its own initializer.
Code
class A { constructor(a: string) { } }
let a: A | null = null;
while (true) {
    const why = a ? 'a' : 'b';
    a = new A(why);
}

Expected behavior: infer why as string
Actual behavior: why inferred as any

Playground Link: https://www.typescriptlang.org/play/#src=class A { constructor(a%3A string) { } }
let a%3A A | null %3D null%3B
while (true) {
    const why %3D a %3F 'a' %3A 'b'%3B
    a %3D new A(why)%3B
}

Related Issues:

The error message means what it says: why does witness itself in its own initializer (in the second iteration of the loop).

@RyanCavanaugh Can you clarify more? why should be string in this case even if it does witness itself, because both branches give string anyway.

This is still broken only with strictNullChecks 🤷♀

@saschanaz SNC matters here because otherwise a isn't a union

@RyanCavanaugh Yeah, but how does that make a potentially not a string?


Negative Ys on Vector layer

I'm drawing rectangles on a map with two layers: a tiled high res image and a vector layer.
When I call (new GeoJSON()).writeFeature(feature), all the Y coordinates in in geometry are negative. I'm guessing that the origin is the top of the map and going downward as -Y gets smaller.
I'm wondering why the Y are negative?
Before upgrading my app from OL2 to OL3, all my Y coordinates were positive (relative to the bottom of the map, going upward as Y gets bigger). When I draw the OL2 features on a OL3 map, they don't appear anywhere. When I draw my OL3 features on my OL2 map, all features appear BELOW the bottom of the image.
Is that due to a settings, or is it a change in the default behaviour from OL2 to OL3?
I can provide more details about the way I construct my maps and obtain the feature coordinates if that helps.

This is probably because you set an incorrect view center, or have a different projection configuration. It's probably better to ask this question on http://stackoverflow.com with the openlayers-3 tag, and provide a JSFiddle with your code.

http://gis.stackexchange.com/questions/203081/direction-of-the-y-axis-in-openlayers


Supports IA/Archive in CreateBucket API and Restore API

1 Supports IA/Archive in CreateBucket API.
---The original CreateBucket does not have the request body, which does not fully implement the PUT Bucket API spec.
2 Add Restore API.
Address some code review comments

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 3 committers have signed the CLA.:white_check_mark: baiyubin:white_check_mark: qixu001:x: qiqi seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


make extension (plugins) cleo script for VSC  (sanny builder)

i wonder if you can make extension for VSC
its a Sanny builder which is cleo editor it don't sound much interested here but as you can say just 10 to 15 million user using this days you can see your self and yea it will be great if it support full functions to edit and create script from VSC to .cs scripts

Hi @Samthapa76 can you please share more details about this issue, if it is related to VS Code
Please create your issue here https://github.com/microsoft/vscode/issues

Since we haven't heard back from you in a while, we assume the issue is resolved. Feel free to create a new ticket if you have any other issues.


CloudStack uses 1st cdrom of VM to mount ConfigDrive ISO

ISSUE TYPE

Bug Report

COMPONENT NAME

ConfigDrive

CLOUDSTACK VERSION

4.14.1

CONFIGURATION

N/A
OS / ENVIRONMENT

ESXi 6.5
CentOS7
SUMMARY

This bug report actually has 2 root cause that solving each of them can solve the bug.

Based on design document of ConfigDrive feature, the iso should be mounted on 2nd cdrom drive of VM but CS don't even create it.

When user tries to attach an ISO to the created VM, CloudStack tries to replace the ISOs but don't answer the esxi question about unmounting iso, so the operation fails.

STEPS TO REPRODUCE

Create a network with ConfigDrive as userdata
Create a VM attached to that network and start it
Attach an ISO to the VM

EXPECTED RESULTS

CloudStack creates the 2nd cdrom drive for VM and mount ConfigDrive ISO on it
or
CloudStack answer the question of ESXi about unmounting ConfigDrive ISO

ACTUAL RESULTS

CloudStack tries to unmount ConfigDrive iso but leave the esxi question unanswered so operation fails.

@aleskxyz is this a regression compared to 4.14.0, that you know?
have you tried this on master?

@aleskxyz is this a regression compared to 4.14.0, that you know?
have you tried this on master?

@DaanHoogland
I see this bug in 4.14.0 too.
I didn't test it on master.

ok, @aleskxyz tnx for reporting

It's not a blocker, but an issue with the proposed design. @aleskxyz are you facing any operation issue due to this? We've tested cloud-init based template in our labs where it works.

It's not a blocker, but an issue with the proposed design. @aleskxyz are you facing any operation issue due to this? We've tested cloud-init based template in our labs where it works.

@rhtyd
As I describe in bug description, this bug prevents attaching iso to VM.

@aleskxyz attaching configdrive ISO to the 2nd drive would involve considerable design changes on vmware end which is something we can target for the next major release - 4.16. As an immediate fix to mitigate this issue, we can work on facilitating a force unmount of the attached configdrive iso - i.e., enable ACS to answer the ESXi question.

Fixed in https://github.com/apache/cloudstack/pull/4907


Syntax highlighting

Would it be possible to add syntax highlighting support for source code blocks?
So that, for example, the following AsciiDoc source would be nicely highlighted in the output:
[source,html]
----

This is some HTML

----

AsciiDoc will convert above snippet to:

<body>
  <p>This is some HTML</p>
  </body>

You'll have to use Eleventy syntax highlight plugin to highlight syntax.

I did try that, but for some reason the HTML is unchanged in the output. The syntax highlight plugin doesn't seem to process it. But I assume it should work, so I’ll give it another go.
Thanks in any case for the nice AsciiDoc plugin!

My bad, Eleventy syntax highlight plugin doesn't solve the problem.
Here in this sample project I have used highlight.js to highlight code on the front end.

Right, so that does the highlighting on the client side. Thanks again!


TextFileReader: The cpu utilization rate is very high, nearly 50%

Expected behavior
Actual behavior
When using TextFileReader to read the log file, the CPU usage rate is very high, nearly 50%. After the file is read, there is also a usage rate of nearly 20%.
Steps to reproduce
System configuration
InLong version: x.y

I took a look at the source code, the while loop inside, there is empty polling for reading, which takes up too much CPU clock, can it be optimized?
I try this code:
@Override     public Message read() {         if (iterator != null && iterator.hasNext()) {             String message = iterator.next();             if (validateMessage(message)) {                 textFileMetric.readNum.incr();                 return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8));             }         }         AgentUtils.silenceSleepInMs(100); // This seems to work, is there a better way ??         return null;     }

public

I took a look at the source code, the while loop inside, there is empty polling for reading, which takes up too much CPU clock, can it be optimized?
    @Override
    public Message read() {
        if (iterator != null && iterator.hasNext()) {
            String message = iterator.next();
            if (validateMessage(message)) {
                textFileMetric.readNum.incr();
                return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8));
            }
        }
        AgentUtils.silenceSleepInMs(100); // This seems to work, is there a better way ??
        return null;
    }

good point @dockerzhang  pls check this


problem on SPAdes installation

Dear author,
I was not able to install STRONG from the auto installer. I followed the manual installation step by step, but still failed on SPAdes ./build_cog_tools.sh . I install the SPAdes from conda, but in the dependency checking it looked at SPAdes/assembler/build_spades/bin/unitig-coverage, which is not part of the SPAdes package.
I have no idea what I can do now and I need you help on installation.
One suggestion, if you could provide a docker image which will be much easier for future users.
Thank you,
Yuxiang

Dear Yuxiang,
We use a particular version of SPAdes with tools allowing us to get high resolution assembly graph. That version of SPAdes is present in the repo and nothing will work well if you try to install some other SPAdes from, for instance, conda.
The auto installer is just a small script which execute all the step from the "manual" installation. Hence, if the auto install doesn't work it's likely the manual one won't.
When you say the auto install didn't work, can you expand on that? Can you share with me the corresponding error message so that I can help solve your issue?
Best,
Seb

Hi Sebastien,
Thank you for your reply. Attached are the logs. I could not figure our why it failed. That is why I install from conda, which as you said is not appropriate.
Honestly, since there are fixed version software, will it be possible to provide a docker image that everyone could easily use and avoid this kind of installation issues?
Best,
Yuxiang
build_cog_tools.log
install_strong.log

Hi Yuxiang,
I can see an issue in the compilation log, it seems like even though the zlib library is installed, you are missing the zlib.h header file. Unsure about what system you're working on but this would be fixed on ubuntu with:
sudo apt-get -y install libbz2-dev libreadline-dev cmake g++ zlib1g zlib1g-dev
Also, I can see that the path to zlib is :
/home/tanyuxiang/.conda/envs/STRONG/lib/libz.so
It looks like it is inside of the conda env. I assume that you installed libz in conda only?
Could you try having libbz2-dev libreadline-dev cmake g++ zlib1g zlib1g-dev installed system wide.
Then you would need to retry that step:
cd <path/to/strong>/STRONG/SPAdes/assembler
./build_cog_tools.sh

I didn't answer you on the docker image the first time, for the simple reason that I don't know how to do that right now and this is not the most urgent thing I need to focus on at the moment. If we can't solve your issue, I may try and find time to do that.
Best,
Seb

run.log
Hi Sebastien,
Thanks for all your reply and I found building a docker will be much easier way for me to solve it.
Now, I build a docker and pass all the check (SnakeNest/scripts/check_on_dependencies.py)
However, when I run your test data, I got two type of errors:

when I unzip the Test.tar.gz, it is truncated (I downloaded twice but all the same situation)
Although the dry run is same as the post with the Step #3 - Strain Decomposition error, I got more errors when I run the actual pipeline (possibly related to the error 1). The log attached.

So, how could I make sure the docker is correct?
Once it is fully checked, I can upload the docker to dockerhub and share it will you.
Best,
Yuxiang

Hi Yuxiang
It's nice you managed to build a docker image which pass the checks.

Hum, strange, I can download/unzip perfectly fine. Either an issue with downloading or with unziping? Here is the md5, can you check yours? I hope the issue is on unzipping, because you can try doing it differently. If you can't manage to download the file, I'm unsure what could be done.
3704340ce2ea5a268dae4cf1a6eaa171  Test.tar.gz

yes, dryrun failing is normal, it is a sequential process and snakemake cannot anticipate what to do without previous steps output. Regarding spades, I had a look at the logs, and I can see that spades is failing though it is unclear why. Could you share/have a look at the file: assembly/spades.log. That's the log of spades and it will tell you why it's failing. If you do have truncated files, it's possible spades stop himself from difference in reads numbers between R1 and R2 files, or just malformed entries. So then it's just a matter of getting correct data.

Well, if when creating a docker image your installation went without issue and the check tells you it's working, then I would say it is correct and works. 100% confidence would be achieved by being able to run 1 of the test dataset.
Thanks for that, that would extremely helpful.
Best,
Seb

Hi, Seb,
Yes, I created a docker image without issue and the check told me it's working.
However, even if I checked the downloaded Test.tar.gz and checked the md5 and it is not truncated this time, the docker run still  ended with an error. (attached)
run.log
I push the docker to docker hub already: https://hub.docker.com/r/yuxiangtan/strong, maybe you can test is yourself as well. (The COG database is not within, users just need to point the path in the config.yaml as you mentioned in the github)
Best,
Yuxiang

Hi Yuxiang,
Cool, nice it worked. Ok, the log tell me it fails at creating a fig with an R script, though real log related to that script are for instance at:
results/Bin_2/tmp/haplotypes_tree.log
Can you tell me what's inside? Most likely a missing library, more troublesome would be an error message from lib version not being the right ones.
Thanks a lot for that, I will have a go at testing it and documenting how to use it.
Best,
Seb

Following are the error message, looks like a more troublesome one.
Warning message:
package 'tidyr' was built under R version 4.0.3
Warning message:
package 'stringr' was built under R version 4.0.5
Registered S3 method overwritten by 'treeio':
method     from
root.phylo ape
ggtree v2.4.1  For help: https://yulab-smu.top/treedata-book/
If you use ggtree in published research, please cite the most appropriate paper(s):
ESC[36m-ESC[39m Guangchuang Yu. Using ggtree to visualize data on tree-like structures. Current Protocols in Bioinformatics, 2020, 69:e96. doi:10.1002/cpbi.96
ESC[36m-ESC[39m Guangchuang Yu, Tommy Tsan-Yuk Lam, Huachen Zhu, Yi Guan. Two methods for mapping and visualizing associated data on phylogeny using ggtree. Molecular Biology and Evolution 2018, 35(12):3041-3043. doi:10.1093/molbev/msy194
ESC[36m-ESC[39m Guangchuang Yu, David Smith, Huachen Zhu, Yi Guan, Tommy Tsan-Yuk Lam. ggtree: an R package for visualization and annotation of phylogenetic trees with their covariates and other associated data. Methods in Ecology and Evolution 2017, 8(1):28-36. doi:10.1111/2041-210X.12628
Attaching package: 'ggtree'
The following object is masked from 'package:tidyr':
expand

Warning message:
package 'ggtree' was built under R version 4.0.3
Warning message:
package 'ggplot2' was built under R version 4.0.5
Warning message:
package 'wesanderson' was built under R version 4.0.5
Error in DataMask$new(.data, caller_env) :
argument "caller_env" is missing, with no default
Calls: ggtree ... mutate.data.frame -> mutate_cols ->  -> initialize
Execution halted
haplotypes_tree.log (END)

Hi, Seb
I run the test data for more than two week already, but it is still running and I found there was no resume option.
Also, I check the server is still running the job but using very little resources.
I am not sure what's happening now and what should I do.
BTW, I need to restart the server in a few days.
Thank you!
Following is the log.
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.

[Mon Sep 20 03:59:45 2021]
rule reads_yaml:
output: samples.yaml
jobid: 5
resources: tmpdir=/tmp
[Mon Sep 20 03:59:46 2021]
Finished job 5.
1 of 121 steps (1%) done
Select jobs to execute...
[Mon Sep 20 03:59:46 2021]
rule coverage:
output: profile/split/coverage.tsv
jobid: 1
resources: tmpdir=/tmp

Hi Yuxiang,
From the logs it seems you've been running with only 1 core, that's something you can change. Then it would be faster :)
There is a resume function. This pipeline is written with snakemake, a workflow management system, which check what's left to run and schedule tasks automatically. So, what's happening exactly? You are relaunching STRONG with the same exact config file and it restart from scratch? That should not happen.... You can try using the option -s -t, which asks snakemake to go through the whole workflow and update timestamps. Since snakemake also check timestamp consistency when deciding what needs to be rerun. Example: file needed for a task is more recent than output of the task, then the task need to be rerun.
I am still looking at that issue with R library and will answer you shortly.
Best,
Seb

Hi Seb,
OK, I thought I set the core number in the config file. I tried 30 core again, and it finished the spade step now. However, it broke at step 1, which I have no idea what I could do.
Following is the key error message and the full log file attached.
Step #1 - Assembly / Binning / COG Annotation
Traceback (most recent call last):
File "/STRONG/bin//STRONG", line 94, in
call_snake(["--snakefile", "SnakeNest/SCogSubGraph.snake"])
File "/STRONG/bin//STRONG", line 83, in call_snake
subprocess.check_call(base_params + extra_params, stdout=sys.stdout, stderr=sys.stderr)
File "/root/miniconda3/envs/STRONG/lib/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['snakemake', '--directory', '/data/Xianjinyuan/tanyuxiang/FMT_mouse-WY/STRONG_all/rerun2', '--cores', '30', '--config', 'LOCAL_DIR=/STRONG', '--configfile=/data/Xianjinyuan/tanyuxiang/FMT_mouse-WY/STRONG_all/rerun2/config.yaml', '--latency-wait', '120', '-k', '--snakefile', 'SnakeNest/SCogSubGraph.snake']' returned non-zero exit status 1.
run.log
Best,
Yuxiang

Hi Yuxiang,
Can you share the file at :  profile/assembly/index.log
That would be where the error message would be stored.
Best,
Seb

Hi, Seb,
I ran a new test on a very small set of simulation test data.
However, I still got error and not able to go through the whole process. (run.log attached)
run.log
Up till now, I only successfully ran the whole pipeline on the provided test data, but all failed in other datasets.
I am not sure what to do now, and I am not sure whether the docker I built is ready for public in this case.
Best,
Yuxiang

Hi Yuxiang,
Sorry for the lack of answer previously, as your issue takes more time to solve I need to secure some on my schedule and never got around to do so.
From a quick look at your log, that is linked to running gtdb. I updated the STRONG not too long ago to deal whith this particular issue of locating gtdb reference genome. I may need to pin version in the conda env, so that these don't occur in the futur.
Please have a go at using latest version of STRONG and tell me how it goes.
Cheers,
Seb

Hi Seb,
I pulled the code and it fixed the gtdb problem
But I got a new problem at the plot_tree_fig step. Log attached.
run.log
Best,
Yuxiang

Hi @Sebastien-Raguideau ,
Any update about this since then?
I updated the previous fixes in docker v1.
Best,
Yuxiang


Manuall installation

can someone pleas help me do this manually, without cloudFormation i want the the installation of this architecture from the start to finish but manually if that makes sense...
please, and thanks.

Hey @RedaElHajjaji,
this will be a looong description.
Do you want to deploy manually the whole environment including the dynamic scaling or which option do you look into?
If, there are quite a things todo.
Best approach would be to go through the Cloudformation templates in the order they're deployed from the root stack.
then check each resource deployed and adapt manually the creation in the different Services affected.
Best,
David

okay, is the "Template structure and deployment workflow" section containe the cloudformation templates in order?

@couchgott can you please help me with it, on where i should start..?

hi,
dorry for the delay but i am vacationing right now.
best is to look into the yaml template files at the root folder of the project.
one is deploying the needed ecr registry and another the s3 bucket to place container images and scripts used for bootstrapping. doing all manually you can also refer to e.g docker hub directly and use the scripts manually depending on your needs. most scripts are used to orchestrate when scaling.
the root template yaml you can use to see which template order is used zo deploy and you can then walk through each nested template to see the needed resources strp by step. hope thid helps. best David

Hey, hope you're doing well
does the root template contain everything that needed to be created ? becasue i see that there are scripts but there are not used ?

Hi,
the scripts are placed at the bootstrap S3 bucket and loaded onto the instances within the bootstrap process. You'll find this at the bbbapp* templates in the UserData Section of the EC2 Instance resource. the UsedData does some prep work and then sideload an bootstrap script. the script then orchestrates the copy and variable handling of each needed script and service and finally installs BBB.
it's a bit cascaded how the actual setup is done.
Best,
David

Hey,
in the setup.sh script there is a typo in line 127, it should be $SCALELITEREGISTRY instead of $SCALEILITEREGISTRY
and thanks

yes you're right. was a late commit :-D

fixed at the last release


perf: Tighten useless information from rendered structures

Currently, a bone that is rendered looks like this:
"structure": {
  ...
  "active": {
    "descr": "Aktiv",
    "type": "select",
    "required": true,
    "params": {},
    "visible": true,
    "readonly": false,
    "unique": false,
    "languages": null,
    "emptyvalue": null,
    "indexed": true,
    "multiple": false,
    "values": {
      "0": "Nein",
      "1": "Ja"
    },
    "sortindex": 8,
  }
  ...
}

but it can also be defined as below, when assuming some defaults for the given attributes:
"structure": {
  ...
  "active": {
    "descr": "Aktiv",
    "type": "select",
    "required": true,
    "indexed": true,
    "values": {
      "0": "Nein",
      "1": "Ja"
    },
    "sortindex": 8,
  }
  ...
}

This omits attributes like visible: true or readonly: false, as they need only be set when explicitly required, so a visible: false-bone is attributes as so, same as a readonly: true bone.
It tightens the amount of data being sent to the client without loosing any information. We should discuss about this.

#1203 relates to this.


CLI Bug

There is a bug while simulating an input if the Step function returns an error. Can be fixed by logging an error when Step returns an error

#5
solved in this pull request


Problem with installation

Running make install after compiling places headers in /usr/local/evhtp/ dir (under ubuntu). Using those global headers causes compilation to fail because evhtp.h includes files assuming they are installed in /usr/local/ dir instead.

 Let me fix.

Oh how lovely. Thank you!

I think this is a short-term fix. What do you think?

It's okay as a short-term fix. It will be much better after general refactor, I'm sure.

Yep.

merged and closing. Thanks for pointing this out!


Use SPDX license identifier

PEP 639 recommends the use of SPDX license identifiers.
At the moment the license key is ignored by flit, see https://github.com/pypa/flit/issues/525. However, it's still possible to replace it now. Once the support in flit is added, the value will be used automatically.
At some point, the table will need be be changed to just a string for full PEP 639 compliance. That requires support for it first though. The change is also pretty simple.
-license = {text = "BSD-3-Clause"}
+license = "BSD-3-Clause"

--
Before the move to pyproject.toml, setup.cfg used BSD-3-Clause as well. https://github.com/pallets/click/blob/8.1.7/setup.cfg#L12

Looks like license is a modified BSD-3 so I am not sure this is correct. @davidism thoughts?

Looks like license is a modified BSD-3 so I am not sure this is correct. @davidism thoughts?

Can you explain what specifically you mean? Github recognizes it correctly as the BSD-3-Clause license.
https://github.com/pallets/click/blob/main/LICENSE.txt
https://spdx.org/licenses/BSD-3-Clause.html

It's not modified, should be standard BSD-3-Clause.

I'm going to close this for now. Once Flit supports license = "string" and license_files = [...], you're welcome to create a PR for that.


ValueError: cannot reshape array of size 0 into shape (0,newaxis)

This might not be the better place for it, but I keep getting this error when adding the predictions and gt:

ValueError: cannot reshape array of size 0 into shape (0,newaxis)

    metric_fn.add(np.array(pred), np.array(gt))
  File "/usr/local/lib/python3.6/dist-packages/mean_average_precision/mean_average_precision.py", line 63, in add
    match_table = compute_match_table(preds_c, gt_c, self.imgs_counter)
  File "/usr/local/lib/python3.6/dist-packages/mean_average_precision/utils.py", line 139, in compute_match_table
    difficult = np.repeat(gt[:, 5], preds.shape[0], axis=0).reshape(preds[:, 5].shape[0], -1).tolist()
ValueError: cannot reshape array of size 0 into shape (0,newaxis)

From the traceback, the issue seems to be happening here:
difficult = np.repeat(gt[:, 5], preds.shape[0], axis=0).reshape(preds[:, 5].shape[0], -1).tolist()
But if perform it manually:
print(pred)
print(gt)
print(np.repeat(gt[:, 5], pred.shape[0], axis=0).reshape(pred[:, 5].shape[0], -1).tolist())

I don't get any error at all:
[[  0.        81.        77.       222.         0.         0.724039]]
[[  0.  83.  72. 184.   0.   0.   0.]]
[[0.0]]

The error message is misleading since this error is related to the num_classes argument of the metrics: for the script to run properly, all classes should be present in the predictions. However, this is certainly a bug: num_classes should rather refer to the classes of the ground truth.
This will fail:
# create metric_fn
metric_fn = MeanAveragePrecision(num_classes=3)#<-see here

# [xmin, ymin, xmax, ymax, class_id, difficult, crowd]
gt = np.array([
    [439, 157, 556, 241, 0, 0, 0],
])

# [xmin, ymin, xmax, ymax, class_id, confidence]
preds = np.array([
    [429, 219, 528, 247, 0, 0.460851],
    [433, 260, 506, 336, 1, 0.269833], #<-see here: only 2 classes present in the preds
])

# add some samples to evaluation
for i in range(10):
    metric_fn.add(preds, gt)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-118-04771a0da08d> in <module>
     15 # add some samples to evaluation
     16 for i in range(10):
---> 17     metric_fn.add(preds, gt)
     18
     19 # compute PASCAL VOC metric

~/anaconda3/envs/rdaw/lib/python3.7/site-packages/mean_average_precision/mean_average_precision.py in add(self, preds, gt)
     61             if preds.shape[0] > 0:
     62                 preds_c = preds[preds[:, 4] == c]
---> 63                 match_table = compute_match_table(preds_c, gt_c, self.imgs_counter)
     64                 self.match_table[c] = self.match_table[c].append(match_table)
     65         self.imgs_counter = self.imgs_counter + 1

~/anaconda3/envs/rdaw/lib/python3.7/site-packages/mean_average_precision/utils.py in compute_match_table(preds, gt, img_id)
    137     img_ids = [img_id for i in range(preds.shape[0])]
    138     confidence = preds[:, 5].tolist()
--> 139     difficult = np.repeat(gt[:, 5], preds.shape[0], axis=0).reshape(preds[:, 5].shape[0], -1).tolist()
    140     crowd = np.repeat(gt[:, 6], preds.shape[0], axis=0).reshape(preds[:, 5].shape[0], -1).tolist()
    141     match_table = {

ValueError: cannot reshape array of size 0 into shape (0,newaxis)

But this is OK:
# create metric_fn
metric_fn = MeanAveragePrecision(num_classes=2) #<-see here

# [xmin, ymin, xmax, ymax, class_id, difficult, crowd]
gt = np.array([
    [439, 157, 556, 241, 0, 0, 0],
])

# [xmin, ymin, xmax, ymax, class_id, confidence]
preds = np.array([
    [429, 219, 528, 247, 0, 0.460851],
    [433, 260, 506, 336, 1, 0.269833], #<-see here: both 2 classes present in the preds
])

# add some samples to evaluation
for i in range(10):
    metric_fn.add(preds, gt)

The error message is misleading since this error is related to the num_classes argument of the metrics: for the script to run properly, all classes should be present in the predictions. However, this is certainly a bug: num_classes should rather refer to the classes of the ground truth.
This will fail:
# create metric_fn
metric_fn = MeanAveragePrecision(num_classes=3)#<-see here

# [xmin, ymin, xmax, ymax, class_id, difficult, crowd]
gt = np.array([
    [439, 157, 556, 241, 0, 0, 0],
])

# [xmin, ymin, xmax, ymax, class_id, confidence]
preds = np.array([
    [429, 219, 528, 247, 0, 0.460851],
    [433, 260, 506, 336, 1, 0.269833], #<-see here: only 2 classes present in the preds
])

# add some samples to evaluation
for i in range(10):
    metric_fn.add(preds, gt)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-118-04771a0da08d> in <module>
     15 # add some samples to evaluation
     16 for i in range(10):
---> 17     metric_fn.add(preds, gt)
     18
     19 # compute PASCAL VOC metric

~/anaconda3/envs/rdaw/lib/python3.7/site-packages/mean_average_precision/mean_average_precision.py in add(self, preds, gt)
     61             if preds.shape[0] > 0:
     62                 preds_c = preds[preds[:, 4] == c]
---> 63                 match_table = compute_match_table(preds_c, gt_c, self.imgs_counter)
     64                 self.match_table[c] = self.match_table[c].append(match_table)
     65         self.imgs_counter = self.imgs_counter + 1

~/anaconda3/envs/rdaw/lib/python3.7/site-packages/mean_average_precision/utils.py in compute_match_table(preds, gt, img_id)
    137     img_ids = [img_id for i in range(preds.shape[0])]
    138     confidence = preds[:, 5].tolist()
--> 139     difficult = np.repeat(gt[:, 5], preds.shape[0], axis=0).reshape(preds[:, 5].shape[0], -1).tolist()
    140     crowd = np.repeat(gt[:, 6], preds.shape[0], axis=0).reshape(preds[:, 5].shape[0], -1).tolist()
    141     match_table = {

ValueError: cannot reshape array of size 0 into shape (0,newaxis)

But this is OK:
# create metric_fn
metric_fn = MeanAveragePrecision(num_classes=2) #<-see here

# [xmin, ymin, xmax, ymax, class_id, difficult, crowd]
gt = np.array([
    [439, 157, 556, 241, 0, 0, 0],
])

# [xmin, ymin, xmax, ymax, class_id, confidence]
preds = np.array([
    [429, 219, 528, 247, 0, 0.460851],
    [433, 260, 506, 336, 1, 0.269833], #<-see here: both 2 classes present in the preds
])

# add some samples to evaluation
for i in range(10):
    metric_fn.add(preds, gt)

Hi, thanks for your feedback.
This issue was fixed in master, but pypi package has not this fix yet. I'll try to rebuild pypi package as soon as I can.

I still have the problem above while having <IP_ADDRESS> (installed with pip install --upgrade git+https://github.com/bes-dev/mean_average_precision.git). Truth and prediction label sets do not equals:
truth_label_set
{0, 1, 2, 4, 5, 6, 8, 9, 10, 11}

pred_label_set
{0.0, 1.0, 2.0, 4.0, 5.0, 6.0, 8.0, 9.0, 11.0}

Moreover, there are addtional labels (3, 7) that are not represented in the test set but are in the training set.
Now, shall I set num_classes=10 (according to the length of truth_label_set) or num_classes=11 (according to the length of pred_label_set), or num_classes=12 (according to the length of all possible labels)?

When will you upload the newest package to pypi?

it is now live! https://pypi.org/project/mean-average-precision/2<IP_ADDRESS>/ Thanks!


ENV/super: always set HOMEBREW_SDKROOT for Xcode builds

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew tests with your changes locally?

The superenv changes have been working well. One last tweak I think needs making, which I've done here, is to always set HOMEBREW_SDKROOT when the Xcode SDK is requested. Without it, it can mean defaulting to the CLT. This only affects 10.13 and earlier.

Thanks again @Bo98!


Installation does not work

Hello, I installed the Nuki plugin in Hoobs and configured it as follows:
{
"platform": "NukiPlatform",
"hostNameOrIpAddress": "192.xxx.x.xxx",
"hostCallbackApiPort": 40506,
"bridgeIpAddress": "192.xxx.x.xxx",
"bridgeApiPort": 8080,
"bridgeApiToken": "xxxxxx",
"bridgeRebootSwitch": true,
"devices": [
{
"nukiId": 0,
"isDoorSensorEnabled": true,
"isRingToOpenEnabled": false,
"isContinuousModeEnabled": false,
"isDoorbellEnabled": false,
"isSingleAccessoryModeEnabled": false,
"unlatchFromLockedToUnlocked": false,
"unlatchFromUnlockedToUnlocked": false,
"lockFromLockedToLocked": false,
"unlatchLock": true,
"unlatchLockPreventUnlatchIfLocked": true,
"leaveOpen": false,
"defaultLockName": "Lock",
"defaultLatchName": "Latch"
}
],
"isApiEnabled": false,
"apiPort": 40011,
"apiToken": ""
}
Unfortunately I don't see any change in the Apple Home Kit.
Do I have to deactivate Home Kit in the Nuki app or does the plugin access it?

Hi Snub, on NukiID you have to put the ID of your Nuki. Instructions on how to retrieve are on the main page of the project.

okay, that was my mistake, now it works, thanks again.  I assumed the first device has ID 0, the next ID1, etc.


firebase_ml_vision multiple app support

Originally was #23049 but I am creating this issue to track non-default Firebase app support in firebase_ml_vision specifically
@bparrishMines

@collinjackson
This issue has been moved to https://github.com/FirebaseExtended/flutterfire/issues/428. Any further collaboration will be done there.


increase consensus batchsize to 1000

Description
To increase consensus batchsize to 1000 in config.yaml
Motivation and Context
This is to increase the performance with pbft consensus
How Has This Been Tested?
Compare the performance results under stress test with various batchsizes: 2 (default), 100, 500, and 1000.
Checklist:

[x] I have added a Signed-off-by.
[x] I have either added documentation to cover my changes or this change requires no new documentation.
[x] I have either added unit tests to cover my changes or this change requires no new tests.
[ ] I have run golint and have fixed valid warnings in code I have added or modified. This tool generates false positives so you may choose to ignore some warnings. The goal is clean, consistent, and readable code.

The continuous integration build process will run make checks to confirm that tests pass and that code quality meets minimum standards. You may optionally run this locally as PRs will not be accepted until they pass.
Signed-off-by<EMAIL_ADDRESS>

Where are the performance results? These would likely be very useful for the community if you're able to share.

Yes, I jumped on this too, out of interest/curious to see some perf numbers!
Specifically for this PR, it will be good to know by how much does the proposed change improve things (even relatively... if that's easier to report on a single system).

With commit: 9777406bffeb68e35f767f89d0e0c0b72626329b dated 6/23/2016
4 peer network with pbft-batch, security and privacy enabled
chaincode: example02
hardware: Lenovo W540
run duration: 10 min and 60 min
TPS measured at SoapUI for invoke for 10 min/60 min runs:
batchsize = 2, 31/13
batchsize = 100, 36/15
batchsize = 500, 47/21
batchsize=1000, 48/21
TPS measured at SoapUI for query for 10 min/60 min runs:
batchsize = 2, 254/250
batchsize = 100, 255/252
batchsize = 500, 252/247
batchsize=1000, 252/249
For commit level cdfbdcd66de75cb832114d8815999189c7f6da9b dated 6/24/2016 under the same test as above,
TPS measured at SoapUI for invoke for 10 min/60 min runs:
batchsize=1000, 72/66
TPS measured at SoapUI for query for 10 min/60 min runs:
batchsize=1000, 252/248

Interesting, thank you very much!
Noting that the biggest improvement is in the SoapUI for invoke,

batchsize = 2, 31/13
batchsize = 100, 36/15
batchsize = 500, 47/21
batchsize=1000, 48/21

and that this is a considerable improvement (from the existing batchsize value of 2!).
So it seems as the biggest 'bang for the batch(size)' ;-), comes from the increase from 2, 100 and 500. From 500 to 1000 does not give us that much, right? (I see 1 extra 'unit')

If you don't mind, I have 3 questions please (but if I'm a pain ;-) and/or these questions do not make sense, then we can leave this/chat on Slack:

The batchsize in Consensus is the number of pre-prepare sends... right? What's the main trade-off/risk using too high a batchsize (for Concensus) is used (is it memory/CPU/network bandwidth/etc...) ? I bet that something 'has to give'? I honestly don't know enough about this, so happy to learn.

What happens if we go above 1000? Were 1200, 1500 or 2000 feasible to try at all? Or are these simply too high and something goes wrong, or is the performance improvement too negligible, or maybe even worse?

Probably related to the answers (1 and 2) above, but I just want to frame my questions:
Given these results I am wondering whether we should set it to 500, as the improvement from 500 to 1000 seems (to me) like an extra 1 TPS (assuming that a high batch size comes with a price) ?

BTW: I'm interested in this also because we were looking at various TCerts batchsize/levels, and it wasn't always clear what's the right thing to do, tbh.

@JonathanLevi ,  your question 1.  batchsize is how many requests are batched together before passing on to consensus. Basically, we're saving on the PBFT protocol messages.

I have another question: for the invoke case, what do you count as a transaction in those metrics?
Do you make sure that the transaction is added to the blockchain? (Of any node. As long as 1 node has it, at least 2f+1 have participated in the ordering of it.)

@tuand27613 Thank you for the prompt reply. This feels like an important change!
Yes, so while I am purposely over-simplifying this, but broadly speaking, does higher batchsize mean?
"Saving on the PBFT protocol messages"
vs.
"Increasing the workload/stress at the chaincode execution point" (tremendously, potentially)

My 2 cents are that given the limited (at this stage) performance information/data we have at the moment for each component in the system, let alone the full end-to-end integration, and in view of the above performance numbers, we should probably try to change to 500 (and not to 1000 right away).
To me it feels as if 500 is already a big improvement, and I suspect that other components of the system (!) might need to be able to keep up/adapt to this new batchsize, more issues will surface that will need addressing, etc. FWIW.
Maybe I'm taking a too simplistic/prudent approach and I'd love to hear/learn/discuss more, especially in case, there are other benefits to having the batchsize of 1000 ?

I would agree that we need more experience with something like this and not just under performance stress but various other failure modes tested as well.

@JonathanLevi  I'd say increased batchsize ==> increase workload at cc execution point and probably at other points in the system as well.
linking to #1338

Thanks again @tuand27613. Yes, 'and probably at other points in the system as well'... I hear you. Updating the above.
For the record, I agreed with @binhn 's 'no magic' number before the link to #1338 ;-)

I did a test with batchsize=500 on the commit cdfbdcd66de75cb832114d8815999189c7f6da9b, and the invoke tps is 64/66 for 10 min/60 min runs compare to 72/66 for batchsize=1000.

@dongmingh, suggest you create a wiki page where you record your results in tabular format.  It would be easier to read and compare  rather than checking comments in a PR.
Also,  is this something to automate and make part of CI ?  Maybe pick a build a day and build up some historical data.

Yes, from the performance perspective, batchsize 1000 performs better then batchsize 500 only in the beginning (first 10 mins) probably because the queue is bigger.
However, I agree that both batchsize of 500 and 1000 achieve the exact same level of performance (at the end of the 60 min runs).

@tuand27613 I will add these data to performance wiki page.

(Not sure if the answer is obvious to everybody else, but I'll repeat the question: https://github.com/hyperledger/fabric/pull/2003#issuecomment-228436159)

@kchristidis I use SoapUI to generate REST API.  SoapUI calculates numbers of transaction and tps.  The tps in the metrics are from SoapUI.
At the end of test, I calculate the values of a and b based on the numbers of invoke transactions sent from SoapUI.  Then I query a and b to see if they match with the calculated values.  Then I send an invoke followed by a query to see if the transactions get executed correctly.

The tradeoff for larger batches is that sending/receiving them takes more time, which delays other protocol packets.  Imagine a 1GB block into a 4-node network: the primary would try to send out 3GB.  Any commit message (for a previous block) would be delayed until this 1GB block has been sent.
Maybe we should look into specifying the batch size in bytes, not transactions.  More difficult to count in the code, easier to reason about everywhere else.

@corecode I agreed size would be a better metric to balance network io. The question is what size is reasonable.  MTU is about 1500 bytes. I think (not sure) databases default to 4k. We would need some work to figure out the optimal balance.
I would suggest for now, if folks are concerned about 1000 tx batch vs 500, then change to 500 and merge that until we can figure out the size in bytes.

batch size of 1000 on <100 tps throughput means >10s latency. This seems suboptimal as a default value.

Hi @vukolic, I agree. There is also an agreement that we can clearly 'do better than 2'.
Shall we proceed with 500 (for now, until we re-visit the metric, resolve other issues, etc.) ?

Hi @vukolic, I agree. There is also an agreement over the 'we can clearly do better than 2' statement. Feels more like something in between.
Shall we proceed with the 500 value (for now, until we re-visit the metric, test & resolve other issues, etc.) ?

I restarted Travis again, but it appears that this build terminates after overflowing the log max size. We really need to look at better approaches for running builds that don't require DEBUG level logging throughout. Possibly we could reserve that for failed builds?

@vukolic any updates regarding @JonathanLevi's question? Still unsure if this should be merged or not

@srdeson @JonathanLevi Sorry, I did not perceive my opinion is blocking. I think 500 is probably ok.

This PR is failing during unit tests
--- FAIL: TestOutstandingReqsResubmission (0.00s)
	obc-batch_test.go:190: All requests should have been executed and deleted after exec

@corecode @jyellick any ideas?

The batch tests generally assume a batch size of 2.  You can see this line is loading the default config.
My guess is you should be able to fix this by changing that line to the following:
config := loadConfig()
config.Set("general.batchsize", 2)
newObcBatch(0, config, omni)

Thanks @jyellick. @dongmingh would you mind changing that line pushing the commit to see if this passes Travis? This also needs to be rebased on master as there's a merge conflict currently. Thanks.

@jyellick @dongmingh Seems this is still failing with the suggested fix?

Thanks @jyellick and @dongmingh, looks to be passing now. LGTM


Use official package for rust-blockchain

The compilation broke because the feature was merged upstream and @gakonst deleted the feat/auto-impl branch in his fork. This PR removes the patch from Cargo.

oops - thank you very much.


Disable "##vso[task.debug]" logs when running locally

Environment
azure-pipelines-task-lib version: 2.11.3
Issue Description

Is it possible to disable the ##vso[task.debug] logs when using this module locally? I am hoping to use some scripts that run on both local machines and in our build pipelines. Our scripts are working fine in both environments but for local use, it would be nice to turn down the output a bit.
I was hoping maybe an environment variable could solve this but haven't found one that works yet. I've tried NODE_ENV=production and SYSTEM_DEBUG=false but neither worked.

I patched this line and used patch-package


Handle a rebooting broker

This implement the behavior observed from a client connected to a VerneMQ server that is being rebooted.
First it will close the connection, then it will reply with {:error, :econnrefused} a couple of times; and then it will finally accept connections again.

Pull Request Test Coverage Report for Build 313

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 83.934%

Totals

Change from base Build 308:
0.2%

Covered Lines:
512

Relevant Lines:
610

💛  - Coveralls


${pageContext.request.contextPath} not resolved

when running http://localhost:8884/sdoc.jsp ${pageContext.request.contextPath} is not getting resolved.
How can I set ${pageContext.request.contextPath} or make it resolvable?

I am facing similar issue, while deploying my war file on tomcat. Do i have to set ${pageContext.request.contextPath} anywhere?

For me, it was (in addition to jstl) tomcat-embed-jasper is not part of the spring-boot-starter-tomcat artifact and has to be added separately.

org.apache.tomcat.embed
tomcat-embed-jasper

I have found that adding a dependency on  compile 'org.apache.tomcat.embed:tomcat-embed-jasper:8.0.15' works when you run the application using bootRun or run the Application.java class in the IDE. But if you build a fat jar of the project then the embedded-jasper dependency actually causes things to break. The error isn't in this project, but a fix for this error still break something else which is bad :/
java.io.FileNotFoundException: Jar URL cannot be resolved to absolute file path because it does not reside in the file system: war:jar:file:/home/james/Sandboxes/property-management/web/build/libs/property-management-web-1.3.3-SNAPSHOT.jar!/lib/swagger-spring-mvc-ui-0.4.jar

@pacey I too am seeing the same problem. Removing org.apache.tomcat.embed:tomcat-embed-jasper fixes the issue.

If I include the org.apache.tomcat.embed:tomcat-embed-jasper dependency in my maven build, spring boot embedded tomcat fails to start up. Do I need to exclude something from the Spring boot jar?

The plan is to remove sdoc.jsp from the webjar and replace with an appropriate html file. We want to remove all of the hassle that including a jsp is causing.

Sounds like a good idea :+1:


how to add attachment in chatterbot

how to add image in chatter-bot chat message

@krishnaraj1993 Maybe you could write some code that checks if there is an image url in the chat bot's response text and displays it if one exists.


Multi Target UWP  to 1809 and 2004

Description of Change

Bugs Fixed

https://github.com/xamarin/XamarinCommunityToolkit/issues/889

Probably Fixes

https://github.com/xamarin/XamarinCommunityToolkit/issues/1129

API Changes
Adds Target Windows 17763 / 1809
Keeps Target Windows 19031 / 2004
Behavioral Changes
works now again on Windows 17763 / 1809
PR Checklist

[x] Has tests (if omitted, state reason in description)
Has already tests
[x] Has samples (if omitted, state reason in description)
Has already samples
[x] Rebased on top of main at time of PR
[x] Changes adhere to coding standard

[ ] Updated documentation

Don't merge yet, doesn't seem to work

I picked the wrong uap version number now it should really target 1809 and 2004

It works now
nuget package under
https://dev.azure.com/xamarin/public/_build/results?buildId=39464&view=artifacts&pathAsName=false&type=publishedArtifacts

OK, so. This fixes the mentioned issues as it will sort out the SDK that needs to be targeted and it will automatically pick up on the shadows not being there for the popups for instance?
And this closes #942 and #1238 ?

As I mentioned in.
https://github.com/xamarin/XamarinCommunityToolkit/pull/1238#issuecomment-829087936
When you Target for Min Windows 1809 Then the Transparent Background won't work because of this.
#if UWP_18362
if (Element.Color == Color.Transparent)
     flyoutStyle.Setters.Add(new Windows.UI.Xaml.Setter(FlyoutPresenter.IsDefaultShadowEnabledProperty, false));
#endif

The 1809 Target won't Compile the Code and therefore it won't be executed which means no Transparent Background for the Popup.

Right, I noticed that compiler directive in there, yet it breaks right now for people targeting different versions if I understand correctly

That's why I did the pull request https://github.com/xamarin/XamarinCommunityToolkit/pull/942
Where it Targets the 1809 But makes the newer properties available. But they need to be guarded by Api Checks

Maybe Multi Targeting and deciding by case by case basis what to do is saver than having runtime crashes.
For example in the develop branch
https://github.com/xamarin/XamarinCommunityToolkit/blob/develop/src/CommunityToolkit/Xamarin.CommunityToolkit/Effects/Shadow/PlatformShadowEffect.uwp.cs
View.ActualSize is used which is not availble under 1809 with compilation for 1809 and 2004 this would immediatly be detected.
Here I mentioned a possible Fix for 1809 with transparent background, But I have to this it First in Release (Linking Mode if it still works).
https://github.com/xamarin/XamarinCommunityToolkit/pull/1238#issuecomment-827402540

I tested the changes and it works for an UWP App in the Debug Mode. But if I Native Compile it (Release) Popup does not work. Although without changes it doesn't work. I have to investigate further what the problem could be. Maybe the wrong Release compilation Settings.

A fun fact I discovered is that the Xamarin.CommunityToolkit Nuget Package and Xamarin.CommunityToolkit Project behave differently in Version Resolution Management.

Original Nuget Targets 19031 / 2004 does not work in UWP App Targeting Min Version of 1809. So a Project using the nuget package doesn't work.
Original Project Targets  9031 / 2004 works in UWP App Targeting Min Version of 1809.  So the Test Project works.

I removed the Release Compilation Crash Fixes from this Pull Request, because they are reviewed in this Pull Request.
https://github.com/xamarin/XamarinCommunityToolkit/pull/1303

Any news about this? Our projects are blocked in XCT 1.0.3 because of this. Could be part of the 1.3.0 release?

@jfversluis Can this be merged? or do I need to make some changes?.
I think this is the safer approach than
https://github.com/xamarin/XamarinCommunityToolkit/pull/942

@inforithmics thank you for keeping this PR up to date. To give a brief heads up we are currently unsure what the best course of action is in terms of supporting an older version of UWP and also the implications of targeting multiple versions.
I plan to investigate this over the next week or so in order to hopefully help us all come to a decision on whether this is something we should do. This also relates to #1684

Changes made in #1684 were considered safer


Fixing Deputy Director Tests

Fixed failing tests after merging in changes to new Deputy Director bio.
Changes
Switched some references of Fuchs to Silberman, but also removed or
changed some tests to not require an exact string match to the deputy
director for the next change.
Testing

run gulp test:acceptance --sauce=false

Review

@jimmynotjim
@sebworks
@anselmbradford

Checklist

[ ] Changes are limited to a single goal (no scope creep)
[ ] Code can be automatically merged (no conflicts)
[ ] Code follows the standards laid out in the front end playbook
[ ] Passes all existing automated tests
[ ] New functions include new tests
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged
[ ] Visually tested in supported browsers and devices
[ ] Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)

:+1:

Pass :+1:


Dynamic Framework (& Carthage support)

Currently, DKChainableAnimationKit does not have a framework target. Instead the classes are in a project shared with an iOS sample app. This setup is prohibits the use of carthage as a depency manager. Also, imho the frameworks classes and the sample project should be separated.
I went ahead and separated everything into two projects. There now is one project for the actual framework and another one for the sample app. I also added a workspace containing both projects.
In addition I updated the .gitignore to also ignore OS X files like .DS_Store and fixes a spelling mistake (DKChainaleAnimationKit.swift --> DKChainableAnimationKit.swift).

Thx for your work, but when I fetch your branch and running the example project.
Error occurs.

Unfortunately, I am unable to reproduce the error. Could you please provide additional information? Did you try to run the example on the simulator or a real device? Which version of Xcode are you using?
For me, building and running the example succeeds for the simulator and also for my iPhone 6 running iOS 8.3.
It would also be helpful to see the actual error message. You should be able to find it in
~/Library/Logs/CoreSimulator/CoreSimulator.log
or
~/Library/Logs/CoreSimulator/[Device UDID]/system.log.
Sorry for the inconvenience.

When I run the iOS example on the simulator. It build successfully but there is nothing in the simulator but a black screen.
When I run the iOS example project on my iPhone6 Plus running iOS 8.3

This is my xcode version

And this is the log when running on the simulator.
~/Library/Logs/CoreSimulator/CoreSimulator.log

May 24 19:10:06 bogon CoreSimulatorService[1168] <Notice>: com.apple.CoreSimulator.CoreSimulatorService 117.15~1 starting.

Thank you very much for the fast response!
The strange thing is, that you seem to see two instances of the Framework Scheme but no iOS Example Project Scheme.
It should look like this:

I went ahead and shared the iOS Example run Scheme. Maybe that fixes the problem? If not, I will download Xcode 6.3 tomorrow to see, if that causes the error. I am using Xcode 6.3.2.

I am really confused with this, Maybe I should merge it first and see whether this can work. I download your repo and checkout to feature/Framework branch everything works well

I merge this successfully but I donot know why there is a warning...

I have merged your PR, but after  carthage update There is no framework in build folder.

Thank you for merging. I will have a look at the issue as soon as I can find the time. I will get back to you soon.


Optional Chaining on budgeting module bug

We need to check NodeJS versions because optional chaining on the budgeting module is causing an error on compile time in the server.
Error example

https://developer.mozilla.org/es/docs/Web/JavaScript/Reference/Operators/Optional_chaining

@mecoccaro you were able to resolve this by using the nvm manager on your computer?

@sofiarm21 Yes!! nvm works perfect, I just changed to node version 14.17.6 and the error is gone.


Diploid assembly

Hi - Thanks for developing such as great tool. I have a question: I am dealing with a highly heterozygous species for which CANU is keeping both haplotypes. I am hesitating between running salsa before (using the assembly graph) or after haploidising the genome. Would that be a problem to map HiC on a diploid assembly, I guess it would dilute the signal. Thanks for your advice.

Hi,
I wound recommend binning contigs into haplotypes and then running SALSA2 individually on each haplotype by mapping all the Hi-C data to each haplotype. There are several ways you could bin contigs. One of it is using purge_haplotigs. We also encountered issues of heterozygosity while using HiC for scaffolding. In this manuscript, we described the way we separated contigs into haplotypes.


Reasoning

Great work! Would you like to write some reasoning for developing this? How does it compare to https://github.com/distribution/distribution ? Or https://alternativeto.net/software/docker-hub/?license=opensource ?

Hi, thanks for asking!
I'm using vex to host docker images for https://github.com/mjl-/mox. A little bit more context at https://github.com/mjl-/mox/commit/a9b2bc8cec74f6590302d0329a0b0a91521d5363. In short: the docker hub announcement and preference for self-hosting. See https://r.xmox.nl/ for a public instance.
I don't use docker much myself. It doesn't help me much since I'm mostly making Go applications that can be reproducibly (cross) compiled into static binaries. Some users of mox do use docker, and I'm OK with providing docker images. But I didn't want to run a registry using docker (seems common), and I only need a minimal feature set. I hoped it would not take too much time to write something that serves a few files by digest. Comparing existing solutions and setting one up and maintaining it also takes time, and is certainly less fun. And I now know more about docker images and registries...
Vex is probably much more minimal than popular alternatives. Some examples:

Very basic authorization mechanism: any authenticated user has full control over all repositories. Good enough if the images in a registry are maintained by a small organization or small set of people. No teams and user groups etc.
Not very configurable or pluggable. E.g. storage is always in the local file system, along with a builtin database. Good enough for small-scale use.

Looking at https://github.com/distribution/distribution, it appears to have a reference implementation and a library. Is this the "go to" registry software for folks hosting their own registries? I'm probably overlooking the instructions to set it up...
From https://alternativeto.net/software/docker-hub/?license=opensource: gitlab (a huge application), harbor (uses: postgresql, redis, and 6 more components, so also big), a "dedicated container registry" that claims to be both no-frills and based on harbor, and potus that is an authorization service for a registry (?). These likely do much more than vex.
As said, I'm not really into the docker ecosystem. There is probably registry software out there that fits my needs. And I may have missed something important about registries (new manifest types? signatures?). I went by the registry specification on the docker website, but I'm now thinking I should have found the OCI spec sooner and use that. I'm open to suggestions. (:

Thank you for taking the time to answer. This would be great content for the README. Docker distribution is quite easy to setup and also has low requirements, but the repository doesn't have all documentation. https://docs.docker.com/registry/deploying/


[MOBILE] headless cms - code bracket image

Please drop size of this image down by 25% and move above text in this section

Can we make the image show up smaller? like maybe 75% of that size?

like this?

Yes, looks great on desktop. On mobile, can we decrease the padding a bit between the header above and the image?

Approved!


Ability to leak secrets store data?

I saw a link posted to https://github.com/aspnet/DotNetTools/blob/dev/src/dotnet-user-secrets/Internal/SecretsStore.cs in slack and looking at it... I wondered if this class should be marked with a NonSerializedAttribute to help prevent someone from serializing/leaking all the secrets (as part of a log message or the like...) or this should be using DP api's? For example, someone could submit a pr and call this tool and leak all the secrets as part of the build server.

This type is only meant for the command line utility and should not be used as API surface. Was someone compiling against this API and calling into it directly?

Tagging @blowdart .
I don't believe there's any concern here - anyone with access to run code on a machine can read the secrets stored there. There's no particular trust required for this - you just need to be able to run code.

And it's not like we expect or even want you to use this on a build server, it's a developer time/developer machine scenario.

That's great and all, but people will use it everywhere esp with global tools :) I'll close this issue, just wanted to bring this issue up.


Simplify procedure name highlighting

Thanks for the New Year's help! 😛

Thanks for the New Year's help! 😛


Refactoring of shared.js: shared2.js proposal

Refactor shared.js to give simpler and more consistent way to write plugins. For example purpose keysocket-vkontakte.js is used.

I see no problems with this, I'm all for it, yeah!  shared2.js will most definitely replace shared.js directly, but perhaps a file name like plugin-ksock.js would be a desirable touch?  (I'm one that enjoys fanciness, lol, plus that's a name that just came into my head.  XD)


Switch to github actions

Currently isort uses .travis to run tests cases, however these tests can be very slow to run, as travis imposes limits on how many run in parallel. This ticket represents the work to move the tests to run on github actions to move past the travis limitation.

Please see PR https://github.com/timothycrosley/isort/pull/1122 for a start on this.

This is now merged in, and active for all testing across OSes. Thanks!


[sct_proseg] Error with absolute path

Error with absolute path but works with relative path (ie from the folder of interest)
syntax:
sct_propseg -i /Volumes/data_shared/boston/brighams/20160204_NIFTI/1.5T_GE/1.5TGE_13332_00014_SAG_FRFSE-XL_T2_POST_TSP_00001.nii -c t2 -ofolder /Users/chgroc/data -init-centerline viewer
error:
Running /Users/chgroc/code/spinalcordtoolbox/scripts/sct_image.py -i /Volumes/data_shared/boston/brighams/20160204_NIFTI/1.5T_GE/1.5TGE_13332_00014_SAG_FRFSE-XL_T2_POST_TSP_00001.nii -o tmp./Volumes/data_shared/boston/brighams/20160204_NIFTI/1.5T_GE/1.5TGE_13332_00014_SAG_FRFSE-XL_T2_POST_TSP_00001_AIL.nii -setorient AIL -v 0
Error of writing permissions on file: tmp./Volumes/data_shared/boston/brighams/20160204_NIFTI/1.5T_GE/1.5TGE_13332_00014_SAG_FRFSE-XL_T2_POST_TSP_00001_AIL.nii
Aborted...

Works without "-init-centerline viewer"!

Indeed, a file needs to be written in the current folder. I'll change that.