Prepare_dataset_vertebral_labeling does not produce bids compliant files

The prepare_dataset_vertebral_labeling script saves images as sub-xxx_<contrast>_mid.nii.gz the contrast should be the last information (sub-xxx_mid_<contrast>.nii.gz) to be bids compliant.
This is needed for the new loader

Thank you for looking into that!
After giving it some thoughts, there is another thing that you will have to do to make the filename BIDS-compliant. In BIDS, every part of the filename must be attached to an entity name to be indexed by pybids and the validator. Otherwise, the files are just ignored.
For example: sub-xxx_rec-mid_<contrast>.nii.gz or sub-xxx_acq-mid_<contrast>.nii.gz.
You can find the list of compliant entity for anat here and their definition here. In that case, I think rec or acq would work.

Hum I get it but this is a problem since the images are straightened. It should appear somewhere I think, to avoid any confusion...
For that matters, this will happen with every 'modified' image ('crop', 'flatten'..) Is there a way to bypass one element? Or maybe we should put that in the bids sidecar?

Sorry, i'm not sure to understand what you mean.
If there is a need to distinguish 2 files based on the filename and those files are not derivatives, then the mid or crop or flatten must be with an entity like rec or acq. So rec-mid, rec-crop, rec-flatten, etc.
If all the files (original, mid, flatten, crop) are in the same dataset and only one of them needs to be loaded, then we will have to implement a new filter in the loader to take care of that. similar to Charley's comment on acq here.
Can you send me a link to one of your dataset so I can better understand the implication? Thanks!

Oh , Ok I get it! I think we can associate that with rec :)
I'll send you the path since this is on the internal server.

As suggested by Julien in slack, another good option would be to create another BIDS dataset for processed data without adding new entity to the filename.
ex:
awesome-data/sub-bob/anat/sub-bob_T1w.nii.gz --> goes into your pre-processing pipeline
awesome-data-processed/sub-bob/anat/sub-bob_T1w.nii.gz --> goes into training

each dataset will be git-annexed into our internal database. And we can version track the source dataset (awesome-data) and the script that enables to generate the preprocessed version.

a potentially preferred alternative would be to do a mix between https://github.com/ivadomed/ivadomed/issues/707#issuecomment-796774510 and https://github.com/ivadomed/ivadomed/issues/707#issuecomment-829209716, to explicit that the input data have been manipulated. Eg:
awesome-data-processed/sub-bob/anat/sub-bob_rec-mid_T1w.nii.gz --> goes into training

Oh! I wasn't aware of this issue, but #787 (which has now become https://github.com/ivadomed/ivadomed/pull/852) modifies the script to name the output files using the rec-mid convention. I'll keep this in mind while working on that PR. :)


fix: correct typo in parser for bomgar

:tada: This PR is included in version 2.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Add and install iptables to provide a foundation for transparent proxy.

Add and install iptables to provide a foundation for transparent proxy.

close


About early stopping during training

Hello, i have a question about training settings in Implementation Details, which say
"Evaluate the performance on the F1 score for CASIAv1, and stop training when there is no improvement for 15 epochs."
Is it right that the you choose the best model based on the TEST dataset results (CASIAv1) ?

Thanks for your attention!
Yes. We select the best model from the best pixel-level F1 result obtained from CASIAv1.

Since we haven't heard back from the original poster for some time, I'll close this issue. Please feel free to reopen it if you need to. Thanks!


feat(snapshot-backfill): introduce state to snapshot backfill

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#19720  👈 (View in Graphite)
#19717
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Add a few YAML-LD slides

@anatoly-scherbakov can we do these in a separate YAML-LD deck specifically for the Thursday group call? There's too much to cover if we include all this in the Wednesday breakout.
It's easy to start a new deck. Just copy Templates/Overview.html into a new folder and rename it index.html (for sanity :wink:).

@BigBlueHat Judging from

the name of the directory (JSON-LD Recharter),
and the number of CBOR-LD slides (11 if my memory serves me),

…I thought that was NOT for a breakout session and, in fact, for the WG meeting. Is that not right? I thought recharter would not be discussed on the breakout.
Thus I actually wanted to ask about where the breakout slide deck will be.

@anatoly-scherbakov sorry for the confusion! The title of the breakout session is exactly what you see there...
Maybe lets make a copy of that and add a -long suffix and add all the extra slides? Or we rename it to WG-day or something. You're call, but let's make a copy and add stuff there.
Thanks!
🎩


[typo] Missing '$FAIRSEQ_ROOT/' in examples/wav2vec/unsupervised/scripts/prepare_text.sh

Before submitting

[ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
[X] Did you read the contributor guideline?
[ ] Did you make sure to update the docs?
[ ] Did you write any new necessary tests?

What does this PR do?
Fixes # (issue).
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

Fixed by f8a7c93440cd925f70979a6082c18f830b39e44b


Add climbing presets + fields

Hi,
I'd love to map more climbing routes according the climbing schema on wiki.
See the discussion in:

#31

This PR
This PR adds the viable minumum to tag climbing crags and climbing routes. I will add more features as needed in future (eg. bouldering, area, alternative tagging for crag and boulder etc).
I add only presets climbing/crag and climbing/route + climbing/route_bottom for start.
I built this PR with dist command and deployed an iD editor with those presets, so it can be tested thorougly.
eg.

https://osm.zby.cz/id-climbing/#id=r17130664
https://osm.zby.cz/id-climbing/#id=r14497828

.
.
Climbing terminology
The taxonomy goes like this - it can be grouped by relations, or just by nearstanding nodes:

Climbing area

Climbing crag

Climbing route
Climbing route (start)
Climbing route (top)   <-- in climbing terminology "top" is end of route

Climbing boulder

Climbing route
Climbing route (start)
Climbing route (top)

Number of items

preset name
tags

types from wiki
#

Climbing area
climbing = area

node,area,relation
<1k

Climbing crag
climbing = crag

node,relation
3k

↳ or
natural = cliffsport = climbing

node,way
7k

Climbing boulder
climbing = boulder

node,relation
1k

↳ or
natural = stonesport = climbing

node,area
1k

Climbing route
climbing = route

node,way
1k

Climbing route (start)
climbing = route_bottom

node
4k

Climbing route (top)
climbing = route_top

node
<1k

Everything should have sport=climbing, no?

Thanks @endolith, very good point. I added the sport=climbing via the addTags property. You can test it in the demo iD editor.

The grade options are always the same?  No YDS in the US?
It complains about some points being on trails or cliffs:

But not others:

And doesn't show the same interface for them:

Thanks for the feedback, @endolith 🙂
ad grades) unfortunately tagging allows for different marks to be assigned for different grade systems. So we have to add a field for each of them in iD. I added the YDS, but in future it would perhaps make sense to create a custom iD field type (like type=multi), which would have better UX (user is not supposed to fill in all of them).
climbing:grade:uiaa=*
climbing:grade:french=*
climbing:grade:saxon=*
climbing:grade:yds_class=*

ad "It complains about some points being on trails or cliffs") Good point. I added vertex type for the route_bottom preset.
ad "But not others") I didn't create a preset for route_top for start, but it is easy now. I added the preset.
Please check it again in the iD demo, eg Cat in the Hat

@tyrasd Thank you very much for such a good review. Also I am glad I learned how to correctly propose fields, eg. that I should search taginfo first 🙂
Changes deployed to my id instance, eg:

crag relation: https://osm.zby.cz/id-climbing/#id=r17130664
crag relation: https://osm.zby.cz/id-climbing/#id=r14497828
crag line: https://osm.zby.cz/id-climbing/#id=w745451490
crag area: https://osm.zby.cz/id-climbing/#id=w743598029
route_bottom without name: https://osm.zby.cz/id-climbing/#id=n2351544812
route_bottom+route both with names: https://osm.zby.cz/id-climbing/#id=n4901371127 (Cat in a Hat)

While editing moreFields I also got an idea how to make them more usable: https://github.com/openstreetmap/iD/pull/10181

Hi @tyrasd - just gently bumping this issue. 🙂 🙏


[General] Apache Airflow DAG to run the new phylopic.py script.

Problem Description

In order to get the new phylopic.py script (see #253) into production, we need to implement a new Apache Airflow DAG that will run the script.
Solution Description

Implement such a DAG.  For examples, see src/cc_catalog_airflow/dags/flickr_workflow.py and src/cc_catalog_airflow/dags/wikimedia_workflow.py.  This DAG should be configured to run the main function from src/cc_catalog_airflow/dags/phylopic.py with the date parameter, once per day.  It should have catchup=False, since we'll ingest the whole collection to start things off, and because we're using the 'modified' date (rather than, e.g., 'uploaded').  The concurrency and max_active_runs parameters should both be 1.
Alternatives

We may replace this DAG with some kind of DAG factory in the future, so it should be considered somewhat temporary.

This is blocked until #253 is merged into master.

@mathemancer Can I work on this, since my work on #253 via PR #276 is almost complete?
(Kindly review that PR.)

@akshgpt7 I think that makes sense.  Please make sure to finish #253 first.

@akshgpt7 Have you been working on this?
If so, please note the change I made in PR #313 (necessary for compatibility with case-insensitive file systems).

Yes, I'm working on this.
Thanks for pointing out the changes!


[FIX] Hotfix for handling responses from n-API

Fixes #138
This is a temporary fix until we get rid of the
non-federated deployment option in general

Closes #138

Changes proposed in this pull request:

this is a hotfix, we will get rid of the non-federation mode in the near future
this just makes up a response body in the expected shape and sticks the n-API response inside
also: didn't add any tests because that would require a complete additional e2e workflow (to run the stack with NB_IS_FEDERATION_API=false)

Checklist
This section is for the PR reviewer

[ ] PR has an interpretable title with a prefix ([ENH], [FIX], [REF], [TST], [CI], [MNT], [INF], [MODEL], [DOC]) (see our Contributing Guidelines for more info)
[ ] PR has a label for the release changelog or skip-release (to be applied by maintainers only)
[ ] PR links to GitHub issue with mention Closes #XXXX
[ ] Tests pass
[ ] Checks pass

For new features:

[ ] Tests have been added

For bug fixes:

[ ] There is at least one test that would fail under the original bug conditions.

:rocket: PR was released in v0.2.2 :rocket:


Allow Docker machine label to be configurable

Currently hardcoded to dockerBuild. We would like to use our own label.
https://github.com/AdoptOpenJDK/openjdk-build/blob/ffbede7222000cf51eb96549b018166d20d11b48/pipelines/build/common/openjdk_build_pipeline.groovy#L685
Either via param or config file.
Related #2045

@AdamBrousseau When you say config file, I assume you're referring to this kind of file?

Yes, but ideally I can point at a file in a different location at build time. Ie. Use my own instead of the default one in the main repo. This goes back to my other issue (#2045) about providing user's own configs. For now, being able to override in the build configs param like I do for the other values like node labels would work.


fix: removed (incorrect) test for symm, trans attributes

Fixes error I made: introducing a test for symm and trans attributes to check that the lemmas were "genuinely" symm/trans lemmas. However the tests had false positives, e.g. #855 (which has been added to the tests).
This short commit removes the tests. They were not present in mathlib3 anyway and the worst that happens if a lemma is incorrectly labelled is that the symm tactic tries and fails with this.
As a warning it may be useful to have a test, but it should have no false positives. For now it seems the best approach is to drop the test.

bors merge


Don't transfer layout outputs to java if they didn't change

See facebook/yoga#483. We should not transfer the layout if the layout didn't change. (using hasNewLayout). This also changes that the lock on the java node is only aquired if needed, and it holds the lock only for the time the values are set and not for the time all it's children are set.

I'm not sure we can directly use hasNewValue as it is not ever set to false in the yoga code. Yoga.c definetly needs some changes as well. There are also times when the padding of a node could potentially have change but not the width which me may or may not want to handle.

We could do the following: We use hasNewLayout just for jni, and keep the value in java as a simple property (no jni callback) and set this value from YGTransferLayoutOutputsRecursive as well. What do you think?

Good idea! I think that should work. React-native depends quite heavily on that value so while Yoga's tests will pass I'll need to run through some internal RN apps to check that nothing unexpected breaks. But I think this idea will probably work.

I also added that we reuse the mHasSetMargin/Padding/Border flags. We make 3 calls more than usual, but as I think that not all node have all of these values set (if at all) we at safe 9 jni calls in the best case.

Further reduces the jni calls by 2 for the edges. By moving from bool to an int flag which stores all set edges. We could even provide a method which does something like TouchFromJni() which sets the hasNewLayout flag and returns all the flags in a single call.

Nice, I like this: can we do the hasNewLayout change in a separate PR though? I ask in case something breaks here we can debug easier and revert a more isolated diff.

@astreet Sure, will put up a separte PR and apply the changes tonight. Thanks!

On the other hand we could provide a ExperimentalFeatureOptimizeJniTransfer which we could enable/disable if anything breaks.

Thanks! I'm fine with this not being an experimental flag :)

I tried landing a much smaller scope version of this earlier which broke a bunch of stuff. I'm going to try to land that first still before landing this but I really like what you have done here. I'll ping you when it is in.

Can you rebase this? 👍


ENH-000085092  The Utility Isolation Trace Web application provided through Local Government Information Model, Solutions is designed for use with water utilities only.  The Gas solution is not available.

Salesforce ID: ENH-000085092
Salesforce Submitter: Brenda McLuskie
Salesforce Submit Date: 2/05/2015 10:47 AM
Salesforce Bug Type: Product
Salesforce Severity: Medium
Repro Data: 0
Work Around:   My customer is a gas company ,  he wants to use the Utilities Isolation Trace tool for his gas data.  The data in the Local Government Information Model  has many similarities,  for example valves,  flow direction, customers effected. He has worked very hard to make the this water template functional with the gas data by changing field names  and rebuilding models.  He would like to see the templates undated to support both water and gas utilities.
Product: ArcGIS for Utilities and Telecom
Functional Category: ArcGIS for Utilities and Telecom
Client Platform: (n/a)
Version Found: 10.2
Planned Version Fixed: (n/a)
Comment: Enhancement added to solutions backlog.

Gas solution release will be available on Feb 18th, 2016


Prevent NoMethodError when api_version is nil in VmImport

When the provider was not yet refreshed (and thus api_version was nil) the validate_import_vm method would raise an exception.

!miq-bot assign @borod108

LGTM

@miq-bot assign @durandom

A small counter intuitive thing, which is probably irrelevant - '4' >= '4.0' is false.

@borod108 I believe that (at least with the RHV provider) the versions always use the long form so this is not an issue. Anyway the point of this PR is to fix the case when it is nil and not malformed :-)

I think that we can close this PR. Solution to this issue was merged https://github.com/ManageIQ/manageiq-providers-ovirt/pull/3

Closed in favor of ManageIQ/manageiq-providers-ovirt#3


Merged scripts into one py-chrome-tools.py

Mode is now selected using 'bookmarks' or 'history' parameter
Updated README
1st step toward adding new functionalities such as
duplicate detection, merging bookmarks, deleting duplicate, old or invalid booksmarks

This seems to be geared toward a massive change you’re planning for this project. You’re welcome to make such changes on your own but I’m not interested in maintaining them. (Alternatively, I’d be happy to accept PRs to add new features to the existing scripts in a mostly-backward-compatible way, such as by passing flags on the command line.)


feat(scene composer): color picker bug fixes

Overview
This change contains color-picker bug fixes. This change is already merged to release-3.x branch.
Verifying Changes
Scene Composer
For scene-composer package changes specifically, you can preview the component in the published storybook artifact. To do this, wait for the Publish Storybook action to complete below.

Click on the workflow details
Select the Summary item on the left
Download the zip file

To run the storybook build locally, you need a local static web server:
npm install -g httpserver
cd <Extracted Zip Directory>
httpserver

Then open the website http://localhost:8080 to run the doc site.
Legal
This project is available under the Apache 2.0 License.

Overview
This change contains color-picker bug fixes. This change is already merged to release-3.x branch.
Verifying Changes
Scene Composer
For scene-composer package changes specifically, you can preview the component in the published storybook artifact. To do this, wait for the Publish Storybook action to complete below.

Click on the workflow details

Select the Summary item on the left

Download the zip file

To run the storybook build locally, you need a local static web server:

npm install -g httpserver

cd <Extracted Zip Directory>

httpserver

Then open the website http://localhost:8080 to run the doc site.
Legal
This project is available under the Apache 2.0 License.


Reduce repeated method calls

Picked up during https://github.com/populationgenomics/production-pipelines/pull/955
We have a bad habit of unnecessarily repeated method calls. Two main offenders are treated here:
The first is Stage B picking up multiple outputs from Stage A. Instead of grabbing both at once:
StageA_Inputs = inputs.as_dict(target, StageA)
in_1 = StageA_Inputs['one']
in_2 = StageA_Inputs['two']

We regularly get each input separately:
in_1 = inputs.as_dict(target, StageA, 'one')
in_2 = inputs.as_dict(target, StageA, 'two')

Seems pretty innocent, but duplicates the method calls, most of which cascade down through continued regeneration of the alignment_inputs_hash. We're probably doing more hashing than a Bitcoin miner, and no $BTC to show for it.
Second example is the repeated generation of the expected_outputs, e.g.
json_path = self.expected_outputs(dataset)['json']
html_path = self.expected_outputs(dataset)['html']
checks_path = self.expected_outputs(dataset)['checks']
...
return self.make_outputs(dataset, data=self.expected_outputs(dataset), jobs=jobs)

Which could be simplified down to
outputs = self.expected_outputs(dataset)
json_path = outputs['json']
html_path = outputs['html']
checks_path = outputs['checks']
...
return self.make_outputs(dataset, data=outputs, jobs=jobs)

Resulting in only a single call to self.expected_outputs instead of 4, and all the resulting non-calls of the hashing algorithm.
I don't know how much time this will save, but we have pipelines that take 40 mins to start up, and this probably isn't helping.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 79.54%. Comparing base (e679675) to head (2a82b12).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #956      +/-   ##
==========================================
+ Coverage   78.42%   79.54%   +1.11%
==========================================
  Files          10       10
  Lines        1794     1794
==========================================
+ Hits         1407     1427      +20
+ Misses        387      367      -20

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Closing this PR, it's substantially mitigated by #958, and muddies the water with additional spurious PRs at this time


lr scheduler state not saved with DeepSpeed plugin

🐛 Bug
When using deepspeed plugin, the learning rate scheduler state is not saved. The key lr_scheduler in mp_rank_00_model_states.pt model file is None. The bug causes the resuming from checkpoint to behave unexpected, where it should continue from previous lr scheduler state but instead lr scheduler start from scratch. This issue happens for most pytorch_lightning release versions, but the issue does not happen for some commits, e.g. commit 86c033572da084fd85d4d4c2d637db18d2d431ec.
To Reproduce
import os
import torch
from torch.utils.data import Dataset, DataLoader
import pytorch_lightning as pl
from pytorch_lightning import LightningModule, Trainer
from pytorch_lightning.callbacks import ModelCheckpoint
from pytorch_lightning.plugins import DeepSpeedPlugin
from deepspeed.ops.adam import FusedAdam
from transformers.optimization import AdamW, get_scheduler

class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):

    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def train_dataloader(self):
        return DataLoader(RandomDataset(32, 64), batch_size=2)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss)
        return loss

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)
        return loss

    def test_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("test_loss", loss)
        return loss

    def configure_optimizers(self):
        no_decay = ["bias"]
        params_decay = [
            p for n, p in self.named_parameters() if not any(nd in n for nd in no_decay)
        ]
        params_nodecay = [
            p for n, p in self.named_parameters() if any(nd in n for nd in no_decay)
        ]
        optim_groups = [
            {
                "params": params_decay,
                "weight_decay": 0.01,
            },
            {"params": params_nodecay, "weight_decay": 0.0},
        ]
        optimizer = torch.optim.SGD(optim_groups, lr=0.1)
        scheduler = get_scheduler("linear", optimizer, num_warmup_steps=10, num_training_steps=100)
        return ([optimizer], [{"scheduler": scheduler}])

def run():
    train_data = DataLoader(RandomDataset(32, 64), batch_size=4)
    val_data = DataLoader(RandomDataset(32, 64), batch_size=4)
    test_data = DataLoader(RandomDataset(32, 64), batch_size=4)

    model = BoringModel()
    checkpoint_callback = ModelCheckpoint(
        dirpath='tests4/checkpoints',
        save_last=True,
        every_n_train_steps=5,
    )
    trainer = Trainer(
        default_root_dir=os.getcwd(),
        gpus=-1,
        limit_train_batches=1.0,
        limit_val_batches=1,
        num_sanity_val_steps=0,
        precision=16,
        accelerator='deepspeed',
        max_steps=50,
        log_every_n_steps=10,
        plugins=[DeepSpeedPlugin(stage=2)],
        weights_summary=None,
        callbacks=[checkpoint_callback, pl.callbacks.LearningRateMonitor(logging_interval="step")],
        #resume_from_checkpoint='tests4/checkpoints/epoch=1-step=49.ckpt',
    )
    trainer.fit(model)
    trainer.test(model, test_dataloaders=test_data)

    # test lr_scheduler state
    model_state_path = "tests4/checkpoints/epoch=1-step=49.ckpt/global_step50/mp_rank_00_model_states.pt"
    model_state = torch.load(model_state_path, map_location="cpu")
    print(model_state["lr_scheduler"])

if __name__ == '__main__':
    run()

Just run python run.py. For latest release version 1.5.6 and many other versions including current master branch, it produces None, meaning lr_scheduler state is not saved. For commit id 86c033572da084fd85d4d4c2d637db18d2d431ec, it produces the correct state:
{'base_lrs': [0.1, 0.1], 'last_epoch': 47, '_step_count': 48, 'verbose': False, '_get_lr_called_within_step': False, '_last_lr': [0.05888888888888889, 0.05888888888888889], 'lr_lambdas': [None, None]}

Do first run with the code exactly as above. Do second run by changing max_steps from 50 to 100, and uncomment #resume_from_checkpoint line. The two runs gives the tensorboard curves of resuming training. For pytorch lightning version 1.5.6 and many others, it produces wrong lr curve:

However, the correct resuming curve (like the one produced with commit id 86c033572da084fd85d4d4c2d637db18d2d431ec) should be

Expected behavior
The lr_scheduler state should be saved properly in order to resume from checkpoint correctly. Could you clarify if we should handle lr_scheduler state loading or saving by ourselves, or it should be properly handled by pytorch_lightning? If it should be handled by pytorch_lightning, do you know how to resolve the bug?
cc @tchaton @SeanNaren @awaelchli

Thanks for reporting this. There could be a bug when assigning the scheduler using the Lightning configure_optimizers hook. It could be that it works when setting it via the config.

@awaelchli Thanks for the reply. Would you elaborate "It could be that it works when setting it via the config"? As temporary workaround, how to set via config?

I'm seeing the same bug with Deepspeed not restoring the lr scheduler.

Ok, I came to the bottom of this. Here is the origin of this bug:
https://github.com/PyTorchLightning/pytorch-lightning/blob/05ed9a201c24e08c2b4d3df4735296758ddcd6a5/pytorch_lightning/strategies/deepspeed.py#L481-L487
If the scheduler is defined by the LightningModule (not via json config file), we set deepspeed_engine.lr_scheduler = None. This is all fine until we want to save a checkpoint, where we have this logic:
https://github.com/PyTorchLightning/pytorch-lightning/blob/05ed9a201c24e08c2b4d3df4735296758ddcd6a5/pytorch_lightning/strategies/deepspeed.py#L742-L746
We exclude the lr_scheduler key from the dict because assume deepspeed.save_checkpoint will save the lr scheduler dict, but since we set deepspeed_engine.lr_scheduler = None this will skip it.
So I think the fix is to modify the save_checkpoint and load_checkpoint to take care of the lr scheduler dict whenever the scheduler is NOT managed by deepspeed internally.

Hey @awaelchli,
Would changing _exclude_keys to ["state_dict", "optimizer_states"] solve the bug ?

Yes I looked into this already, already discussed with @SeanNaren he agrees

Hello @tridao @eelxpeng
If you have the time, could you quickly check my fixes in this branch here #11322 if it works with your use case? I have checked with your repro-example, just wanted to let you have a look.

Updated the repro-example since there is some issue with it.
@awaelchli I checked the your fix, and it works great! See the following curves obtained. Thank you very much!

Thanks for checking, glad we were able to make progress here. I will open the PR for review.


Left handed "fast" scrolling

Is your feature request related to a problem? Please describe.
Scrolling by dragging the scrollbar thumb (to quickly scroll to an older date/year for instance) is only available when using the phone right handed
Describe the solution you'd like
Ability to switch sides of the scrollbar thumb
Describe alternatives you've considered
using other hand :)

I'm unlikely to provide this option because, like many other UI elements, the layout is conventional according to the text direction of the current locale. For left-to-right, the drawer is on the left, the overflow menu is on the right, keys are on the left, values on the right, etc. And I don't want to micromanage the layout beyond that convention.
But I'm curious, how do you manage with other apps?
By the way, i'm surprised you intend to grab the thumb with one hand. I know i can't do that, even with the thumb on the "convenient" side ^^

I don't,  and it's been an annoyance for a long time in all the gallery apps ;).
But I completly understand now.
I am used to this pattern of scrolling with a thumb - I use Niagara launcher so it's a very natural movement for me.
The tbumb is a little bit too far,  but still reachable by one hand.
Thank you for the reply, feel free to close this.


Adding leading zeros to string don't get saved to database

Laravel Version
10.40.0
PHP Version
8.2
Database Driver & Version
MySQL 8.0.30 on CentOS 7 (x86)
Description
Let's say we have an eloquent model Test, that has a string property prop. A database record exists where prop contains the string value "1234". We load this record using eloquent models, and update the value to contain leading zeros like this:
$model->prop = "01234";
$model->save();

In this case, the database value will not be updated. This is caused by the following code in src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php on line 2103:
return is_numeric($attribute) && is_numeric($original)
            && BigDecimal::of($attribute)->isEqualTo($original);

Since the string values do pass is_numeric and their number counterparts are equal, the property isn't considered dirty and the field isn't updated. These string values might represent totally different things in the database and shouldn't be considered equal in all cases.
I understand changing this might impact existing codebases, but is there any way we can provide an easier escape hatch than overwriting HasAttributes::originalIsEquivalent (as overwriting core functionality tends to break in the future)?
Steps To Reproduce

use Illuminate\Database\Eloquent\Model;

/**
 * @property string $prop
 */
class Test extends Model {
  protected $fillable = ['prop'];
}

$test = new Test(['prop' => '1234']);
$test->save();

$test->prop = '01234';
$test->save(); // saving doesn't happen because prop isn't considered dirty

This seems to be happening because no explicit cast has been defined and it has to guess the data type. I believe you can workaround this by explicitly casting to string:
class Test extends Model {
    protected $fillable = ['prop'];
    protected $casts = ['prop' => 'string'];
}

See: https://github.com/laravel/framework/blob/10.x/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php#L2092-L2094


How to re-trigger a failed peribolos workflow

As a peribolos-as-a-service user, I would like to have a facility to re-trigger the run of the peribolos workflows, as it runs on a pushed commit, I would like to re-run, as the failed/missed workflows should have succeeded so that I can verify really there is a problem or this was just a flake.

/assign


Fix compilation of projects using cc-rs on macOS

After commit 0b0b182f8d2e12a1558d12dc5aea15bc815c703b, cc-rs now invokes
the compiler found by CMake, for example
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc

instead of
/usr/bin/cc

The latter uses the default SDK sysroot root as configured with xcode.
The latter requires either specifying --sysroot /path/to/sysroot (which
is what CMake does) or it falls back to using the SDKROOT environment
variable. Without a sysroot the compiler can't find standard includes.
This patch sets SDKROOT as environment variable to be set during cargo
invocation, to the same value that CMake passes to the compiler. That
way cc-rs won't need any additional flags.
cc #152

This makes sense to me.
However I think this fix may not work for the scenario where cmake is configured for cross-compiling and the "Host build" option is set for a crate. Is it possible to adjust the fix to consider this scenario?

However I think this fix may not work for the scenario where cmake is configured for cross-compiling and the "Host build" option is set for a crate.

Can you elaborate how setting SDKROOT may break cross-compiling?
This environment variable is only used by the Apple clang compilers, so if cross-compiling to a different macOS version and architecture, then CMAKE_OSX_SYSROOT should be set correctly by the user of CMake, I think (or rather, cmake sets it correctly based on the deployment target).

Cross compiling would work fine. The issue appears when you are cross-compiling, but mark one crate as "Host build", i.e. don't cross-compile this specific crate. In that case the CMAKE_HOST variables would probably have to be used to get the correct information.
On the other hand I'm not sure why we have the Host build option at all. I think users could just use build scripts to achieve build time code generation, but maybe I'm overlooking something.

Ahh, yes, now I get it.  I changed the patch.
Regarding the host build feature: We have this use-case that a CMake project consists of a library and a separate tool (a little compiler). Traditionally with CMake, cross-compiling in such a scenario requires building the entire project for the host first - in order to get a tool binary that can run - and then building the project another time, into a different prefix, in order to get the library for the target. The second build then may need some extra trickery to import the tool from the first build, to make it appear as seamless in usage as if it were a host == target build.
With cargo this is fortunately not necessary, we can invoke cargo build -p libproj --target=foo and cargo build -p tool  or - for pure Rust projects - use the tool via proc macros and it'll be correctly build for the host arch.
The corrosion host build feature brings that convenience to a CMake "wrapper" and eliminates the need for two cmake builds.
That said, in Slint - where we use this - we could also do without and go two a two-stage build system. I think eventually we may need that anyway for other reasons. So this is merely my explanation of the past, I have no objections if you want to remove this feature again in a future Corrosion release :-)

Ahh, yes, now I get it.  I changed the patch.

Thanks. This new fix will not be sufficient for the case where you cross-compile from e.g. Linux to Mac, but I guess that is pretty exotic anyway, so I'm fine with merging this PR as is.

Regarding the host build feature: [...]

Thanks for elaborating on that! We actually have a similar use-case at work (we also cross-compile), where we have a tool that generates code. But since it's written in 100% Rust, we just do add_custom_target(tool COMMAND ${CARGO_COMMAND} build --target-dir [...]) without involving Corrosion at all.
For now, I don't mind leaving the host build feature in though.


Update lora_config_bloom.json

The lora_r should be changed to r since the parameters in LoraConfig is r

The lora_r should be changed to r since the parameters in LoraConfig is r

Should not change the config.


Networked facialExpressions

QA test
Prerequisites

Please use https://github.com/highfidelity/hifi/pull/12584 to test (only required for the person using the app, the other person should be on the same protocol of course).
This test requires two avatars (Interface clients) in the same domain.
Tester A make sure to use an avatar with proper blendshapes. It works the best with the Artemis avatar which should be available from market.

Test

Tester A: run the facialExpressions script
Tester B: watch User A's face closely and remember their Neutral expression.
Tester A: Set your view to mirror view to see your own face (shortcut 2 View->Mirror ). (For comparance, be aware that the view is mirrored)
Tester A: Change your facial expression to each one of the following emotions and confirm with Tester B:
'H' = smile
'J' = laugh
'K' = flirt
'L = sad
'V' = angry
'B' = fear
'M' = disgust
'N' = neutral
Tester B: Make sure that you see the emotion change every time.
Tester A: Stop the script.
Tester A: Look at yourself, your face should have returned to neutral.
Tester B: Confirm that Tester A their face returned to their neutral expression

@thoys Does this need separate QA from the PR that has the support for blend shapes, or can this test be run on that PR?

QA approved per https://github.com/highfidelity/hifi/pull/13338


Using Python to connect to cosmosDB throws Authorization header doesn't confirm to the required format

Hi,
I am facing some issue when I use your library to connect to my cosmosDB in python3.7. I have setup a private cosmosDB database and it has a private endpoint in my private subnet. I use one VM i the private subnet to run below python codes.
>>> import azure.cosmos.cosmos_client as cosmos_client
import azure.cosmos.errors as errors
import azure.cosmos.http_constants as http_constants
>>> >>> >>> import os
>>> client = cosmos_client.CosmosClient('https://thermo-breast-cancer-cosmos-account.table.cosmos.azure.com:443',  'KrZklvMYWBcIaujH2APJES8eAsjCZDaUSyVNec6VUi28aeUWwn4io48aFPqlyuYFnhtXq3hlPzhZIuUQ4tCaww==')
>>> for z in client.list_databases():
...     print(z)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/azure/core/paging.py", line 122, in __next__
    return next(self._page_iterator)
  File "/usr/local/lib/python3.7/site-packages/azure/core/paging.py", line 74, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_query_iterable.py", line 98, in _fetch_next
    block = self._ex_context.fetch_next_block()
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_execution_context/execution_dispatcher.py", line 110, in fetch_next_block
    raise e
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_execution_context/execution_dispatcher.py", line 102, in fetch_next_block
    return self._execution_context.fetch_next_block()
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_execution_context/base_execution_context.py", line 75, in fetch_next_block
    return self._fetch_next_block()
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_execution_context/base_execution_context.py", line 163, in _fetch_next_block
    return self._fetch_items_helper_with_retries(self._fetch_function)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_execution_context/base_execution_context.py", line 135, in _fetch_items_helper_with_retries
    return _retry_utility.Execute(self._client, self._client._global_endpoint_manager, callback)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_retry_utility.py", line 75, in Execute
    result = ExecuteFunction(function, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_retry_utility.py", line 130, in ExecuteFunction
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_execution_context/base_execution_context.py", line 133, in callback
    return self._fetch_items_helper_no_retries(fetch_function)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_execution_context/base_execution_context.py", line 116, in _fetch_items_helper_no_retries
    (fetched_items, response_headers) = fetch_function(self._options)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_cosmos_client_connection.py", line 344, in fetch_fn
    lambda _, b: b, query, options, **kwargs
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_cosmos_client_connection.py", line 2385, in __QueryFeed
    result, self.last_response_headers = self.__Get(path, request_params, headers, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_cosmos_client_connection.py", line 2217, in __Get
    **kwargs
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_synchronized_request.py", line 218, in SynchronizedRequest
    **kwargs
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_retry_utility.py", line 73, in Execute
    result = ExecuteFunction(function, global_endpoint_manager, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_retry_utility.py", line 130, in ExecuteFunction
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/cosmos/_synchronized_request.py", line 158, in _Request
    raise exceptions.CosmosHttpResponseError(message=data, response=response)
azure.cosmos.exceptions.CosmosHttpResponseError: Status code: 401
{"odata.error":{"code":"Unauthorized","message":{"lang":"en-us","value":"Authorization header doesn't confirm to the required format. Please verify and try again.\r\nActivityId: a0610060-e318-4b68-b08d-362f71671963, documentdb-dotnet-sdk/2.11.0 Host/64-bit MicrosoftWindowsNT/6.2.9200.0\nRequestID:a0610060-e318-4b68-b08d-362f71671963\n"}}}

As you see , it gives me some error regrading mismatch hear authorization. Perhaps, I need to tweek the codes but I do not know how to fix this error?
Any help would be appreciated.

Duplicate of https://github.com/Azure/azure-sdk-for-python/issues/13225

Hi Steve,
This issue has been resolved for me so you may close or delete the threads.
Thanks,
Mahdi
From: Steve Faulkner
Sent: Thursday, 20 August 2020 6:54 AM
To: Azure/azure-cosmos-python
Cc: Bazarganigilani; Author
Subject: Re: [Azure/azure-cosmos-python] Using Python to connect to cosmosDBthrows Authorization header doesn't confirm to the required format (#203)
Duplicate of Azure/azure-sdk-for-python#13225
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.


[nvcc][fix] Fix compilation problems with HIP_nvcc toolchain

Fix several compilation problems with HIP_nvcc toolchain (CUDA Toolkit 11.8):

HIP-Basic/matrix_multiplication

main.hip

HIP-Basic/moving_average

main.hip
moving_average_vs2017.vcxproj
moving_average_vs2019.vcxproj
moving_average_vs2022.vcxproj

HIP-Basic/multi_gpu_data_transfer

main.hip

HIP-Basic/opengl_interop

opengl_interop_vs2022.vcxproj

HIP-Basic/shared_memory

main.hip

HIP-Basic/streams

streams_vs2017.vcxproj
streams_vs2019.vcxproj
streams_vs2022.vcxproj

HIP-Basic/warp_shuffle

main.hip

Whats the status on this? It would be great to get these changes, so that we don't have to do any duplicate work.


APNs supports notifications without a topic, but Pushy does not

Hi,
When using certificate-based authentication, the APNs API allows clients to omit the apns-topic header, and will use the bundle ID associated with the certificate as the default topic - see Listing 2 at the bottom of this page.
However, Pushy does not allow the construction of notifications without specifying a topic. Passing a null value for the topic to the constructor of SimpleApnsPushNotification gives the following exception:
Exception in thread "main" java.lang.NullPointerException: Destination topic must not be null.
        at java.util.Objects.requireNonNull(Objects.java:228)
        at com.turo.pushy.apns.util.SimpleApnsPushNotification.<init>(SimpleApnsPushNotification.java:195)
        at com.turo.pushy.apns.util.SimpleApnsPushNotification.<init>(SimpleApnsPushNotification.java:73)
        at ApnsTest.main(ApnsTest.java:46)

and the javadoc on the ApnsPushNotification interface states that the getTopic() method must not return null.
Would you be willing to remove the requireNonNull restriction to support notifications without a topic? I'd be happy to submit a PR for this.
Thanks,
Tom

Would you be willing to remove the requireNonNull restriction to support notifications without a topic?

I certainly understand your point, but this is a deliberate design decision and I don't think we're really open to changing it. In the immediate term, always having a topic avoids some branches internally and makes expectations of callers more consistent. In the future, always having a topic will allow us to do multi-credential clients (we'll need the topic to figure out which credentials to use if multiple certificates are in play).
I appreciate the willingness to contribute, but let's leave this one as-is.
Thanks!


Version 2.5.0 uses Python 3.10 features

Please rewrite or update requirements

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."


Make dragonfly working on dreamhost

removed optimization for compatibility with ImageMagick 6.3.7 of Debian Lenny

@markevans what do you think?


track client IP in analytics plugin

This is actually already sent by the plugin.

keeping this open to verify where the issue is.

@thibaultCha can we close this?


Remove jsVersion option

It is currently used only by Firefox as explained in the doc and Firefox is removing it internally, so it doesn't make much sense to keep the option around for too long

Your Pull Request would be welcomed!


⚠ Samsung has degraded performance

In 90246fd, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2474 ms

Resolved: Samsung performance has improved in c9302d2.


Multiple IMSICs with multiple VS interrupt files for a hart may bring huge resource overhead

Background: There are discussions on this topic in the smmtt community:
https://lists.riscv.org/g/tech-smmtt/topic/multiple_imsic_for_a_hart/106345304
As described by Vedvyas Shanbhogue on the floor 91:

Many implementation choices exist such as physically instantiating multiple IMSICs, shared IMSIC logic with separate interrupt files.

Currently, the smmtt spec only describes the first implementation, which is to instantiate multiple IMSICs physically.
This implementation may bring a problem. In RISCV-V Server SoC specification it is required that "IMSIC MUST support at least 5 VS-mode interrupt files".
1 IMSIC implements at least 5 VS int files; 2 supervisor domains, i.e. 2 IMSICs, need to implement 10 VS int files; And 4 supervisor domains, i.e. 4 IMSICs, need to implement 20 VS int files. If there are more SDs, the number of VS files instantiated in a hart will be greater. And for Server SoC, the number of harts will also be very large. If this is implemented, the logic resources and area overhead will be very exaggerated.
But generally, the number of virtual harts (in VMs or TVMs) running on per physical hart is only 4~5. The number of virtual harts that a physical hart can carry is very limited. In the multiple SDs implementation of SERVER-level SOC, each physical hart has so many VS interrupt files (by instantiating IMSIC for each SD), but only a small number of VS interrupt files are actually used, resulting in a large waste of logic resources and area.
This may be a big problem in actual implementation. Therefore, the second implementation scheme, that is, sharing IMSIC logic with multiple sets of S/VS files for multiple SDs, seems more feasible. In this scheme, we can configure which VS files are exposed to active SD or deactive SD in RDSM. In this way, some VS files can be shared among multiple SDs to reduce the defects of excessive resource and area overhead brought by the first implementation scheme.
Is it possible to add description and supplement to the second implementation option in smmtt spec to guide the actual development? It seems that new register functions need to be introduced.
Please refer to the content of the floor 85, 91, 92, 93, 94, 95 of the discussion page linked above.
Please also help correct me if my understanding is wrong in any way, thanks!

If your hardware has multiple physical IMSIC instances, there is no requirement that they have the same features, as long as each instance is described accurately in the DT/ACPI. For example, you could have one IMSIC for the primary SD with 5 VS-mode interrupt files, and 3 IMSICs for the other SDs each with 0 or 1 VS-mode interrupt file. Each SD sees its own IMSIC in the DT/ACPI, and does not know anything about the IMSICs of other SDs.
If your hardware can move VS-mode interrupt files between S-mode IMSIC groups, then a static allocation of the VS-mode interrupt files would look just like what I described above. You would use some platform-specific method (e.g. custom CSR) to configure the IMSICs at boot to match what is described in each SD's DT/ACPI, and the RDSM would be otherwise unaware of this configurability. So this could be implemented without any changes to the spec, though there may be some benefit to standardizing this IMSIC configuration interface.
If you want to dynamically allocate VS-mode interrupt files to SDs, then the software/hardware is more complicated. You would pretend that each IMSIC instance has the maximum number of VS-mode interrupt files in the DT/ACPI. Then, you would need to trap writes to hstatus.VGEIN to do the actual allocation, possibly context switching some other SD's existing VS-mode IMSIC context to/from RAM. This would require runtime support from the RDSM, so if we want to support this use case, it would be good to standardize the register interface.
Do you have any suggestions for what the interface for configuring the IMSIC would look like? Should this be developed in conjunction with the AIA spec?

Thanks for your reply！
Combined with your description, please let me summarize the current several implementation ideas.
Implementation 1: Using multiple physical IMSIC instances, without hardware support static allocation of the VS interrupt files

If your hardware has multiple physical IMSIC instances, there is no requirement that they have the same features, as long as each instance is described accurately in the DT/ACPI. For example, you could have one IMSIC for the primary SD with 5 VS-mode interrupt files, and 3 IMSICs for the other SDs each with 0 or 1 VS-mode interrupt file. Each SD sees its own IMSIC in the DT/ACPI, and does not know anything about the IMSICs of other SDs.

But there are two suspicious points that need to be clarified first.

The point is as described in the below quotation.
The smmtt spec v0.2.0 does not specify whether the number of VS interrupt files in each physical  IMSIC instance can be inconsistent. But according to RISCV-V Server SoC specification "IMSIC MUST support at least 5 VS-mode interrupt files", we will understand that at least 5 VS interrupt files are required for each active SD. So each physical IMSIC instance should have at least 5 VS interrupt files. So the point seems to contradict RISCV-V Server SoC spec.

there is no requirement that they have the same features

The point is as described in the below quotation.
This requires OS ecosystem support. It is need to promote OS to add supervisor domain dimension in interrupt management.

Each SD sees its own IMSIC in the DT/ACPI

Please help explain the above doubts before determining the feasibility of this Implementation.
Implementation 2: Using multiple physical IMSIC instances, with hardware support static allocation of the VS interrupt files once

If your hardware can move VS-mode interrupt files between S-mode IMSIC groups, then a static allocation of the VS-mode interrupt files would look just like what I described above. You would use some platform-specific method (e.g. custom CSR) to configure the IMSICs at boot to match what is described in each SD's DT/ACPI, and the RDSM would be otherwise unaware of this configurability. So this could be implemented without any changes to the spec, though there may be some benefit to standardizing this IMSIC configuration interface.

This implementation still needs to solve the above two suspicious points first. And I personally think that the hardware may not consider implementing the function of statically allocating VS interrupt files between multiple physical IMSIC instances.
Implementation 3: Using single physical IMSIC but multiple sets of S/VS interrupt files, with support dynamic allocation of S/VS interrupt files to SDs

If you want to dynamically allocate VS-mode interrupt files to SDs, then the software/hardware is more complicated. You would pretend that each IMSIC instance has the maximum number of VS-mode interrupt files in the DT/ACPI. Then, you would need to trap writes to hstatus.VGEIN to do the actual allocation, possibly context switching some other SD's existing VS-mode IMSIC context to/from RAM. This would require runtime support from the RDSM, so if we want to support this use case, it would be good to standardize the register interface.

My understanding here is a little different from yours. Use a single physical IMSIC instead of multiple physical IMSIC instances.
I don't have any more ideas about the specific implementation details of dynamic allocation. However, from the previous discussions on this topic in the smmtt community, I think there is already a good implementation direction. Please let me quote some key descriptions. We can continue to discuss based on this.
https://lists.riscv.org/g/tech-smmtt/topic/multiple_imsic_for_a_hart/106345304

from Alvin Chang on floor # 89:
If we arrange "single" IMSIC's S- and VS- interrupt files as:
{ S[1], G[1], G[2], ......, G[i] }, { S[2], G[i+1], G[i+2], ......, G[j] }, ... { S[x], G[y+1], G[y+2], ......, G[z] }
Then msdcfg.SDICN selects one of these groups in the IMSIC.
It seems this does not violate AIA, but I notice a constraint in AIA section 3.1:
The number of guest interrupt files an IMSIC has for virtual harts is exactly GEILEN ......
IIUC, it means the maximum G[z] is G[63] in my example, for RV64.
That means each { S[1], G[1], G[2], ......, G[i] } group should not have all 63 VS-files in that IMSIC,
Otherwise, other supervisor domains won't get any available VS-files.

from Siqi Zhao on floor # 92:
We could work around this by not grouping S-mode interrupt files and VS-mode interrupt files in the hardware. We could say msdcfg.SDICN only selects the active S-mode interrupt file, and let the TSM select the active VS-file exactly as how a normal hypervisor does, i.e. using the hstatus.VGEIN bit. This is OK since the TSM is in the TCB.
In order to isolate the trusted set of VS-mode files from untrusted set of VS-mode files, a M-mode bit mask could be introduced, 'msdvsifcfg' maybe. Each bit in it corresponds to a VS-mode file, a set bit means that VS-mode file should only be used by SDs. An attempt to access when not executing SDs raises an illegal instruction exception.

from Siqi Zhao on floor # 94:
My idea is that there is only one set of VS-mode interrupt file. Each of the files could be designated as active or inactive. By setting msdvsifcfg, RDSM exposes VS files suitable for the current SD. Then, the TSM sets hstatus.VGEIN to select among the VS-mode interrupt file exposed for the current SD. If for any reason, hstatus.VGEIN is set to the 'hidden' VS file, there should be an exception.
The responsibility is divided up. RDSM exposes the VS files in chunks for each SD, then TSM in a SD selects among the VS files in its chunk.

from Alvin Chang on floor # 95:
IMHO, set to the 'hidden' VS file should be avoided.
According to Priv. Spec. section "Hypervisor Guest External Interrupt Registers (hgeip and hgeie)":
The least-significant bits are implemented first, apart from bit 0. Hence, if GEILEN is nonzero, bits GEILEN:1 shall be writable in hgeie, and all other bit positions shall be read-only zeros in both hgeip and hgeie.
It seems the indexes of available VS files should be contiguous.
Since each IMSIC only has up to 63 physical VS-files, we can format msdvsifcfg as:
msdvsifcfg[11:6] - "number" of selected VS files (also GEILEN for this supervisor domain)
msdvsifcfg[5:0] - "start index" of selected VS files
For example:
- msdvsifcfg[11:6] = 6
- msdvsifcfg[5:0] = 7
It means that 6 IMSIC's physical VS files are selected to this supervisor domain, and the files are indexed as 7 ~ 12.
Base on this selection, when hstatus.VGEIN = 1, it actually selects IMSIC's physical VS file 7, and hstatus.VGEIN = 2 actually selects physical VS file 8 , etc.
So we still stick to the Priv. Spec. about GEILEN.

Regarding your implementation 1 doubts - the spec says:

The supervisor domain interrupt controllers connected to a hart may not be identical. For example, each supervisor domain IMSIC may support a different number of external interrupt identities for S- and/or VS-level, and the number of guest interrupt files supported by each supervisor domain IMSIC may not be identical.

So from a SD spec perspective, each IMSIC may support differing number of guest interrupt files. If a platform implementation is also aiming for Server SOC spec compliance for the OS running as a Supervisor Domain then the HW has to comply with those requirements of  "IMSIC MUST support at least 5 VS-mode interrupt files". Other Supervisor Domains on the platform which are not aiming to satisfy Server SOC spec compliance may still have a lower number of guest interrupt files (or use a RDSM-managed way of sharing guest interrupt files as Samuel suggested).

Hi @rsahita

Other Supervisor Domains on the platform which are not aiming to satisfy Server SOC spec compliance may still have a lower number of guest interrupt files

In fact, what I care about is targeting the application scenarios that I think are the most likely.
Confidential virtual machines are applied on Server SOC. And on the platform all Supervisor Domains are aiming to satisfy Server SOC spec compliance. So each IMSIC MUST support at least 5 VS-mode interrupt files.
But generally, the number of virtual harts (in VMs or TVMs) running on per physical hart is only 4~5.  Each physical hart has so many guest interrupt files (Since each IMSIC has non-shareable guest interrupt files), but only a small number of guest interrupt files are actually used, resulting in a large waste of logic resources and area.
So I think it would be better to include the details of the sharing guest interrupt files solution into the spec. This would satisfy the requirement that all SDs OS supports SERVER while avoiding wasting resources and area.

When considering the support for setting different numbers of VS interrupt files for different Supervisor Domians, I was confused by the definition and implementation of some parameters (e.g. GEILEN, VGEIN, hgeie, and hgeip)
The GEILEN parameter is described in the specification as follows
For any one physical hart, the maximum number of virtual harts that may directly receive guest external interrupts is limited by GEILEN.
This parameter appears to describe the total number of VS interrupt files for a physical hart(regardless of how many S-domains are running on that physical hart);
In the SMMTT specification, VGEIN is described as follows
The hstatus. VGEIN selects a guest interrupt file in the selected IMSIC and vsiselect and vstopei CSRs operate on the registers of the corresponding guest interrupt file in the selected IMSIC.
VGEIN selects a VS interrupt file in an IMSIC on a physical hart. But for the first VS interrupt file of a certain domain, it may actually be the nth VS interrupt file on this physical hart. Should the VGEIN be assigned to the actual VS interrupt file number instead of the VS interrupt file number from a certain domain perspective?
My question is, should the parameters related to VS interrupt files (e.g. GEILEN, VGEIN, HGEIE, and hgeip) be assigned from the perspective of the domain or from the perspective of the physical hart under the smmtt specification?
Many Thanks！

@jimingchen111 see https://github.com/riscv/riscv-smmtt/issues/113#issuecomment-2505112460

Hi @rsahita , I understand what you mean. It can be left unchanged or based on your opinion. Thank you!


Identifier's methods signature

Greetings
Tell me please, why methods of Identifier interface accept reflect.Value? There is no places where it is used how I can see.
https://github.com/bxcodec/faker/blob/master/uuid.go#L24

You can search that keyword in Github, to see where it used.

https://github.com/bxcodec/faker/blob/627121e0bd014927f980e231ba34eebc41d9cf18/uuid.go#L10-L21
https://github.com/bxcodec/faker/blob/627121e0bd014927f980e231ba34eebc41d9cf18/faker.go#L205-L206

This is legacy codes, I think it's since 2017 if you have a better approach, feel free to discuss it here.


Added setting to bypass SSL validation and removed custom HAP build

I appreciate the pull request. However, the version of html agility pack (from nuget 1.4.9) has an issue that causes the entire process to crash due a to a stack overflow exception. This is a problem internal to Html Agility Pack (see more here https://code.google.com/p/abot/issues/detail?id=77&can=1&q=hap). I submitted the patch to the official repo but there is no one maintaining Html Agility Pack anymore. That's why there is a custom Hap dll embedded in the Abot nuget package.
The tests that you removed are the ones that test for this condition. Due to that I cannot except this pull request.

Thanks for the info!
I will check on our end if we can change version.
In the meantime, I looked at the issue and the problem still resides in 1.4.9 as you stated. However did you try using HtmlDocument.OptionAutoCloseOnEnd = true? I was able to load the files you linked in your issue (it takes some time), maybe that could be an alternative.

That option did not help. However after diving into the options I did find one that helped...
HtmlDocument.OptionFixNestedTags = true
Commit message: Removed dependency on patched HtmlAgilityPack.dll file. Now using nuget version 1.4.9 with option [HtmlDocument.OptionFixNestedTags = true] to solve stack overflow problem. The only possible bad side effect is that HAP will now be able to modify the html source to fix nested tag errors which means that the source downloaded by Abot might not be exactly what the server is sending.
Thanks

Good to know!!
Thanks

I was able to get my local build working with the solution above but the build server would still throw a stackoverflow exception. Had to roll back the change:(


Obfuscation bug fixes/improvements, ScriptBlockLogging bypass update

My periodical obfuscation/logging PR:
Fixes:

PowerShell Beta.9 switched the 'powershell' process to be named 'pwsh' which caused some issues. Should work now for either.
launcher_vbs failed with obfuscation, just needed some VBScript quote escaping. fixes #765
preobfuscation error - not sure how/when this issue appeared, but preobfuscate commands were failing due to a path issue. Fixed here.

Updates:

PowerShell install - no longer need to manually download debs on Debian/Kali, woo!
Invoke-Obfuscation - updated to latest version
ScriptBlockLogging bypass - Improved to work on all versions of PS (it had broken in a recent version of PS v5.1)

Good Job @cobbr this nice

@cobbr thanks again man! Tested this out and both issues are resolved.


Android GLES2, error trying to load S3TC textures not present in the APK.

Godot version:
Godot 3.1 Stable

OS/device including version:
Linux Mint 18.2 Cinnamon 64 bit
Intel Core i5-7500T CPU
Intel HD Graphics 630 GPU
Android 4.4.4 Nexus 7 2012

Issue description:
When making Android export, if Import S3TC and Import ETC are selected in project settings->vram compression, only ETC is included in the APK file. However on running on the tablet, it attempts to load non-existent S3TC files, and has loads of errors and doesn't work.
Steps to reproduce:
Create blank ES2 project, add a box with a spatial material and add a texture to albedo. Set vram compression to import S3TC and ETC (defaults). Run on android device -> error.
However, it is possible to get it to work by removing S3TC from the import list (i.e. import ETC only), delete the .import folder to force reimport, then re-exporting. This time the android build does not attempt to load non-existent S3TC, and loads the ETC correctly.
Unfortunately the desktop build does not work with only ETC selected, and you get lots of warnings about the S3TC versions not being available, so you have to manually change this and redo the imports when you want to make an android export.
Minimal reproduction project:
s3tc_android_bug.zip
ADB Logcat:

04-22 18:40:21.760 8675-8689/com.lawnjelly.s3tc W/GodotView: creating OpenGL ES 2.0 context :
04-22 18:40:21.800 8675-8689/com.lawnjelly.s3tc I/godot: OpenGL ES 2.0 Renderer: NVIDIA Tegra 3
04-22 18:40:21.880 482-498/? I/ActivityManager: Displayed com.lawnjelly.s3tc/org.godotengine.godot.Godot: +1s254ms
04-22 18:40:22.430 8675-8689/com.lawnjelly.s3tc E/godot: ERROR: Condition ' !f ' is true. returned: ERR_CANT_OPEN
04-22 18:40:22.430 8675-8689/com.lawnjelly.s3tc E/godot:    At: scene/resources/texture.cpp:493:_load_data() - Condition ' !f ' is true. returned: ERR_CANT_OPEN
04-22 18:40:22.430 8675-8689/com.lawnjelly.s3tc E/godot: ERROR: Failed loading resource: res://.import/Board.png-45a35c08fcd69cecff72ce41392a754f.s3tc.stex
04-22 18:40:22.430 8675-8689/com.lawnjelly.s3tc E/godot:    At: core/io/resource_loader.cpp:285:_load() - Method/Function Failed, returning: RES()
04-22 18:40:22.430 8675-8689/com.lawnjelly.s3tc E/godot: ERROR: Failed loading resource: res://Board.png
04-22 18:40:22.440 8675-8689/com.lawnjelly.s3tc E/godot:    At: core/io/resource_loader.cpp:285:_load() - Method/Function Failed, returning: RES()
04-22 18:40:22.440 8675-8689/com.lawnjelly.s3tc E/godot: ERROR: res://Spatial.tscn:3 - Parse Error: [ext_resource] referenced nonexistent resource at: res://Board.png
04-22 18:40:22.440 8675-8689/com.lawnjelly.s3tc E/godot:    At: scene/resources/resource_format_text.cpp:440:poll() - res://Spatial.tscn:3 - Parse Error: [ext_resource] referenced nonexistent resource at: res://Board.png
04-22 18:40:22.440 8675-8689/com.lawnjelly.s3tc E/godot: ERROR: Condition ' err != OK ' is true. returned: RES()
04-22 18:40:22.440 8675-8689/com.lawnjelly.s3tc E/godot:    At: core/io/resource_loader.cpp:208:load() - Condition ' err != OK ' is true. returned: RES()
04-22 18:40:22.440 8675-8689/com.lawnjelly.s3tc E/godot: ERROR: Failed loading resource: res://Spatial.tscn

More Info:
Having a look in the android exporter, exporter.cpp:
virtual void get_preset_features(const Ref<EditorExportPreset> &p_preset, List<String> *r_features) {

	String driver = ProjectSettings::get_singleton()->get("rendering/quality/driver/driver_name");
	if (driver == "GLES2") {
		r_features->push_back("etc");
	} else if (driver == "GLES3") {
		r_features->push_back("etc2");
		if (ProjectSettings::get_singleton()->get("rendering/quality/driver/fallback_to_gles2")) {
			r_features->push_back("etc");
		}
	}

This looks like to me in android export GLES2 it is maybe choosing to only export etc (which makes sense as there is no need to export S3TC as well, as EXT is mandated in GLES2). However the code that chooses which texture to load also needs to make the same decision, this is probably where the bug lies.
The specific circumstances which may lead to it are : the android device (Nexus 7) supports S3TC and EXT. However, the android studio emulator does not support S3TC (according to OpenGL caps viewer), and it does not make this attempt to load S3TC, and the bug does not occur on the emulator (it goes straight to load ETC and succeeds).
A example way to prevent this, rather than e.g. relying on only the hardware caps of the device, is to write have e.g. a set of bitflags for which compressed texture formats are present in the export, and not attempt to load ones that are not present. I haven't examined this bit of the code.
e.g.
UNCOMPRESSED = 1
S3TC = 2
ETC = 4
APK has 4
Hardware caps is 7
AND them together to get ones available and supported.

I think I'm having a similar issue. I also managed to reproduce this issue with a simple project. My setup is this: I create a simple GLES2 project, create a spatial material with the image texture on it, I export to iOS, install on the phone and get the error pasted below. Basically all I see is the black screen. If I change project to GLES3, then it works but there's different types of issues with GLES3 on mobile.
This is the error I see.
2019-06-05 11:52:08.526124+0300 export-tester[16002:5966467] OpenGL ES 2.0 Renderer: Apple A10 GPU
OpenGL ES 2.0 Renderer: Apple A10 GPU
2019-06-05 11:52:08.623126+0300 export-tester[16002:5966467] [MC] System group container for systemgroup.com.apple.configurationprofiles path is /private/var/containers/Shared/SystemGroup/systemgroup.com.apple.configurationprofiles
2019-06-05 11:52:08.648576+0300 export-tester[16002:5966467] [MC] Reading from public effective user settings.
2019-06-05 11:52:09.148913+0300 export-tester[16002:5966467] **ERROR**: Failed loading resource: res://icon.png
2019-06-05 11:52:09.148943+0300 export-tester[16002:5966467]    At: core/io/resource_loader.cpp:285:_load() - Method/Function Failed, returning: RES()
**ERROR**: Failed loading resource: res://icon.png
   At: core/io/resource_loader.cpp:285:_load() - Method/Function Failed, returning: RES()
2019-06-05 11:52:09.148993+0300 export-tester[16002:5966467] **ERROR**: res://materials/spatialmaterial_test.tres:3 - Parse Error: [ext_resource] referenced nonexistent resource at: res://icon.png
2019-06-05 11:52:09.149007+0300 export-tester[16002:5966467]    At: scene/resources/resource_format_text.cpp:440:poll() - res://materials/spatialmaterial_test.tres:3 - Parse Error: [ext_resource] referenced nonexistent resource at: res://icon.png
**ERROR**: res://materials/spatialmaterial_test.tres:3 - Parse Error: [ext_resource] referenced nonexistent resource at: res://icon.png
   At: scene/resources/resource_format_text.cpp:440:poll() - res://materials/spatialmaterial_test.tres:3 - Parse Error: [ext_resource] referenced nonexistent resource at: res://icon.png
2019-06-05 11:52:09.149027+0300 export-tester[16002:5966467] **ERROR**: Condition ' err != OK ' is true. returned: RES()
2019-06-05 11:52:09.149047+0300 export-tester[16002:5966467]    At: core/io/resource_loader.cpp:208:load() - Condition ' err != OK ' is true. returned: RES()
**ERROR**: Condition ' err != OK ' is true. returned: RES()
   At: core/io/resource_loader.cpp:208:load() - Condition ' err != OK ' is true. returned: RES()
2019-06-05 11:52:09.149277+0300 export-tester[16002:5966467] **ERROR**: Failed loading resource: res://materials/spatialmaterial_test.tres
2019-06-05 11:52:09.149287+0300 export-tester[16002:5966467]    At: core/io/resource_loader.cpp:285:_load() - Method/Function Failed, returning: RES()
**ERROR**: Failed loading resource: res://materials/spatialmaterial_test.tres
   At: core/io/resource_loader.cpp:285:_load() - Method/Function Failed, returning: RES()
2019-06-05 11:52:09.149408+0300 export-tester[16002:5966467] **ERROR**: res://scenes/main_test_scene.tscn:3 - Parse Error: [ext_resource] referenced nonexistent resource at: res://materials/spatialmaterial_test.tres
2019-06-05 11:52:09.149452+0300 export-tester[16002:

I have a fix for this, it is quite simple.
The problem occurs because although the android exporter only exports ETC, there may still be a reference to S3TC files in the .import file for the textures (if you are using S3TC on desktop). If the Android device does not support S3TC, it does not attempt to load the file and there is no problem. If however it does support S3TC it attempts to load the file and fails, and exits the game.
There are several ways of solving it:

Remove the reference from the .import file
Prevent the loader from attempting to load S3TC
Pretend there is no hardware support for S3TC on Android devices

I have gone for option 3 as it was easiest for me to implement.
The change is in rasterizer_storage_gles2.cpp, line 5593:
Old version:
config.float_texture_supported = config.extensions.has("GL_ARB_texture_float") || config.extensions.has("GL_OES_texture_float");
config.s3tc_supported = config.extensions.has("GL_EXT_texture_compression_s3tc") || config.extensions.has("WEBGL_compressed_texture_s3tc");
config.etc1_supported = config.extensions.has("GL_OES_compressed_ETC1_RGB8_texture") || config.extensions.has("WEBGL_compressed_texture_etc1");
config.pvrtc_supported = config.extensions.has("IMG_texture_compression_pvrtc");
config.support_npot_repeat_mipmap = config.extensions.has("GL_OES_texture_npot");

Fixed version:
config.float_texture_supported = config.extensions.has("GL_ARB_texture_float") || config.extensions.has("GL_OES_texture_float");
config.s3tc_supported = config.extensions.has("GL_EXT_texture_compression_s3tc") || config.extensions.has("WEBGL_compressed_texture_s3tc");
config.etc1_supported = config.extensions.has("GL_OES_compressed_ETC1_RGB8_texture") || config.extensions.has("WEBGL_compressed_texture_etc1");
config.pvrtc_supported = config.extensions.has("IMG_texture_compression_pvrtc");
config.support_npot_repeat_mipmap = config.extensions.has("GL_OES_texture_npot");

// Special override for running under android.
// The Android exporter (android exporter, exporter.cpp, get_preset_features())
// specifically prevents s3tc being written to the APK, and this makes sense
// as ETC is mandated in GLES 2.0. However the .import file still contains a reference to the s3tc
// file, and any Android device that support s3tc will attempt to load the resource and fail,
// causing exit.
// An simple way to prevent this circumstance is to pretend that hardware support does not exist
// for S3TC on all android devices, which prevents the attempt at loading.
// An alternative method maybe to remove the reference to the s3tc in the .import file.
#if ANDROID_ENABLED
config.s3tc_supported = false;
#endif

If any of you guys who use git can put this in (or one of the other alternatives if you prefer) that would be great (I'll try and learn how to use git!!). :smile:

I think I'm having a similar issue. I also managed to reproduce this issue with a simple project. My setup is this: I create a simple GLES2 project, create a spatial material with the image texture on it, I export to iOS, install on the phone and get the error pasted below. Basically all I see is the black screen. If I change project to GLES3, then it works but there's different types of issues with GLES3 on mobile.

It does sound similarish. I don't have any iOS devices so I can't investigate really but can you look inside the exported IPA file and see what versions (if at all) of the png it has in the .import folder? If you can open the logo.png.import file in text editor it should tell you what files it will try to load. I hope this works I have no experience with iOS.
If there is no png file in the .import folder (maybe you would need to reimport the texture as lossless in the Godot IDE?) then perhaps adding ETC textures to your project might get it exported. Sorry I would like to help more.

I have spent a bit more time researching this issue and although my above fix addresses the problem in my case, I am now suspecting it may be exposing a wider logic flaw in the whole resource exporting / loading system in Godot (unless the logic is already correct, and this is a simpler bug). To recap, some possible fixes:
1. Remove the reference from the .import file
2. Prevent the loader from attempting to load S3TC
3. Pretend there is no hardware support for S3TC on Android devices
I am now tending towards feeling that in order to address this and future bugs of this type with other resources it would be better to implement (1) or (2). Exploring option 1, it appears that the main export happens in this function:
EditorExportPlatform::export_project_files
Where it queries the exporter for a feature list of formats that are desired for export (which may not include s3tc for example). It then only exports those that are on the feature list.
However I think it is just copying the .import file word for word into the exported project, containing references to files that have not been exported. At this point perhaps a function could be written to alter the text of the .import file and remove the references to features that are not on the feature list? Godot could also perhaps flag up a warning if you were trying to export a resource with no features supported.
This would be a bit more involved than the quick fix of method 3, but would be a generic solution that would be easier to maintain in the long run, and may prevent other similar bugs occurring in future. I would feel hesitant to make this change myself, it is really something that should be examined by guys familiar with the resource loader, and needs discussion.
Of course anything messing directly with text is prone to more bugs, so saving the feature list with the exported project (is this done already?) and method 2 may be more attractive in practice.
So at this point I hesitate to make a pull request for method 3, as it is a quick fix rather than addressing the underlying issue - unless it would spur more interest. Thoughts anyone?

Fixed by #32255.


Fixing commit 8393d87

On commit 8393d87, condition (this.statusCode === 0) line 539 will always be false as this.statusCode is set to 200 by default. Same for this.body line 541 that is an empty object by default. Then the error message was always set to 'The socket request failed.'.
Following code set a more explicit message.

Good catch @acekat.  In practice, the status code some could be 0 -- specifically, the rare case where the socket thinks it's still connected, but the server can't be reached.  But as you discovered, we're defaulting it to 200 whenever it's falsey, which is not the intention.  Looks like this:
this.statusCode = responseCtx.statusCode || 200;

really needs to be:
this.statusCode = (typeof responseCtx.statusCode === 'undefined') ? 200 : responseCtx.statusCode;

and I'm also guessing that
if (this.statusCode === 0) {

is meant to be:
if (this.statusCode !== 0) {

Thank you for your answer @sgress454 !
Would this last commit be better ? I'm keeping the 'The socket request failed.' message in case the statusCode is 0.
Also, I'm stringifying the body to have better log, but it would be safer to use isaacs/json-stringify-safe.


Bug: Perfmon requires Computer Name to remotely poll

I've uninstalled tabmon and removed postgres, reinstalled using all defaults, as that was default issue resolution on previous access denied service start issues.
Now, I get the following:
2016-11-21 11:02:05,407 [5] INFO TabMon.Config.TabMonConfigReader - Loading TabMon user configuration..
2016-11-21 11:02:05,438 [5] DEBUG TabMon.Config.TabMonConfigReader - Loading database configuration..
2016-11-21 11:02:05,438 [5] INFO TabMon.Config.TabMonConfigReader - Connecting to results database..
2016-11-21 11:02:05,720 [5] DEBUG TabMon.Helpers.HostnameHelper - Successfully resolved 'edwtab2.cac.washington.edu' to 'edwtab2'.
2016-11-21 11:02:05,735 [5] INFO TabMon.Config.TabMonConfigReader - Successfully loaded TabMon config options! [Host='Primary\edwtab2', Writer='Database Writer (Postgres Driver)', PollInterval='60', TableName='countersamples']
2016-11-21 11:02:05,735 [5] INFO TabMon.TabMonAgent - Initializing TabMon..
2016-11-21 11:02:05,735 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Loading persistent performance counters from C:\Program Files (x86)\TabMon\Config\Counters.config..
2016-11-21 11:02:05,735 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Loading performance counters from C:\Program Files (x86)\TabMon\Config\Counters.config..
2016-11-21 11:02:05,735 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Successfully validated 'Config\Counters.config' against 'Resources\CountersConfig.xsd'.
2016-11-21 11:02:05,735 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Loading Perfmon counters..
2016-11-21 11:02:10,345 [5] ERROR TabMon.TabMonAgent - Failed to initialize counter sampler using counters from configuration file: The network path was not found.

Aborting..
2016-11-21 11:02:10,345 [5] FATAL Topshelf.Runtime.Windows.WindowsServiceHost - The service did not start successfully
Topshelf.TopshelfException: The service did not start successfully (returned false).
   at Topshelf.Runtime.Windows.WindowsServiceHost.OnStart(String[] args)
2016-11-21 11:02:10,345 [1] ERROR Topshelf.HostFactory - The service terminated abnormally
System.NullReferenceException: Object reference not set to an instance of an object.
   at TabMon.TabMonAgent.Dispose(Boolean disposing)
   at TabMonService.TabMonServiceBootstrapper.Dispose(Boolean disposing)
   at TabMonService.TabMonServiceBootstrapper.Dispose()
   at Topshelf.Builders.ControlServiceBuilder`1.ControlServiceHandle.Dispose()
   at Topshelf.Runtime.Windows.WindowsServiceHost.Dispose(Boolean disposing)
   at System.ComponentModel.Component.Dispose()
   at System.ServiceProcess.ServiceBase.Run(ServiceBase[] services)
   at System.ServiceProcess.ServiceBase.Run(ServiceBase service)
   at Topshelf.Runtime.Windows.WindowsServiceHost.Run()
   at Topshelf.HostFactory.Run(Action`1 configureCallback)

Any assistance in getting tabmon installed and working is appreciated.

Hi Travis,
It looks like TabMon is failing to resolve the hostname correctly.
Can you try changing the Host entry in TabMon.config to point to the IP address (instead of the name edwtab2) and let me know if it works?  If it does, it will confirm my theory and I can check in a fix for this.
Let me know how it goes!

updated to use IP address instead of hostname in tabmon.confg, same failure.
`2016-11-22 07:11:59,437 [5] INFO TabMon.Config.TabMonConfigReader - Loading TabMon user configuration..
2016-11-22 07:11:59,547 [5] DEBUG TabMon.Config.TabMonConfigReader - Loading database configuration..
2016-11-22 07:11:59,547 [5] INFO TabMon.Config.TabMonConfigReader - Connecting to results database..
2016-11-22 07:12:00,140 [5] DEBUG TabMon.Helpers.HostnameHelper - Successfully resolved '<IP_ADDRESS>' to 'edwtab2'.
2016-11-22 07:12:00,172 [5] INFO TabMon.Config.TabMonConfigReader - Successfully loaded TabMon config options! [Host='Primary\edwtab2', Writer='Database Writer (Postgres Driver)', PollInterval='60', TableName='countersamples']
2016-11-22 07:12:00,172 [5] INFO TabMon.TabMonAgent - Initializing TabMon..
2016-11-22 07:12:00,172 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Loading persistent performance counters from C:\Program Files (x86)\TabMon\Config\Counters.config..
2016-11-22 07:12:00,187 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Loading performance counters from C:\Program Files (x86)\TabMon\Config\Counters.config..
2016-11-22 07:12:00,187 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Successfully validated 'Config\Counters.config' against 'Resources\CountersConfig.xsd'.
2016-11-22 07:12:00,187 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Loading Perfmon counters..
2016-11-22 07:12:04,812 [5] ERROR TabMon.TabMonAgent - Failed to initialize counter sampler using counters from configuration file: The network path was not found.
Aborting..
2016-11-22 07:12:04,812 [5] FATAL Topshelf.Runtime.Windows.WindowsServiceHost - The service did not start successfully
Topshelf.TopshelfException: The service did not start successfully (returned false).
at Topshelf.Runtime.Windows.WindowsServiceHost.OnStart(String[] args)
2016-11-22 07:12:04,859 [1] ERROR Topshelf.HostFactory - The service terminated abnormally
System.NullReferenceException: Object reference not set to an instance of an object.
at TabMon.TabMonAgent.Dispose(Boolean disposing)
at TabMonService.TabMonServiceBootstrapper.Dispose(Boolean disposing)
at TabMonService.TabMonServiceBootstrapper.Dispose()
at Topshelf.Builders.ControlServiceBuilder1.ControlServiceHandle.Dispose()    at Topshelf.Runtime.Windows.WindowsServiceHost.Dispose(Boolean disposing)    at System.ComponentModel.Component.Dispose()    at System.ServiceProcess.ServiceBase.Run(ServiceBase[] services)    at System.ServiceProcess.ServiceBase.Run(ServiceBase service)    at Topshelf.Runtime.Windows.WindowsServiceHost.Run()    at Topshelf.HostFactory.Run(Action1 configureCallback)`

It looks like we have a bug here.. both the IP and your longer DNS entry are being resolved and then overly shortened down to simply "edwtab2", which Perfmon can't connect to; we need to preserve the domain suffix information for cases like this.
We can probably turn around a fix for this fairly quickly.. I'll keep you posted here.

Hi Travis, Hi Jmangue
I currently experience the same issue on a distributed environment where TabMon is installed on another server as Tableau Server. Did you find a solution to this problem?
2016-12-13 10:32:28,683 [5] INFO TabMon.Config.TabMonConfigReader - Loading TabMon user configuration..
2016-12-13 10:32:28,715 [5] DEBUG TabMon.Config.TabMonConfigReader - Loading database configuration..
2016-12-13 10:32:28,715 [5] INFO TabMon.Config.TabMonConfigReader - Connecting to results database..
2016-12-13 10:32:28,871 [5] DEBUG TabMon.Helpers.HostnameHelper - Successfully resolved 'sedcatab0010.emea.ebs.corpintra.net' to 'sedcatab0010'.
2016-12-13 10:32:28,871 [5] INFO TabMon.Config.TabMonConfigReader - Successfully loaded TabMon config options! [Host='Primary\sedcatab0010', Writer='Database Writer (Postgres Driver)', PollInterval='60', TableName='countersamples']
2016-12-13 10:32:28,871 [5] INFO TabMon.TabMonAgent - Initializing TabMon..
2016-12-13 10:32:28,886 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Loading persistent performance counters from C:\Program Files (x86)\TabMon\Config\Counters.config..
2016-12-13 10:32:28,886 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Loading performance counters from C:\Program Files (x86)\TabMon\Config\Counters.config..
2016-12-13 10:32:28,886 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Successfully validated 'Config\Counters.config' against 'Resources\CountersConfig.xsd'.
2016-12-13 10:32:28,886 [5] DEBUG TabMon.CounterConfig.CounterConfigLoader - Loading Perfmon counters..
2016-12-13 10:33:10,919 [5] ERROR TabMon.TabMonAgent - Failed to initialize counter sampler using counters from configuration file: The network path was not found.
Aborting..
2016-12-13 10:33:10,919 [5] FATAL Topshelf.Runtime.Windows.WindowsServiceHost - The service did not start successfully
Topshelf.TopshelfException: The service did not start successfully (returned false).
at Topshelf.Runtime.Windows.WindowsServiceHost.OnStart(String[] args)
2016-12-13 10:33:10,919 [1] ERROR Topshelf.HostFactory - The service terminated abnormally
System.NullReferenceException: Object reference not set to an instance of an object.
at TabMon.TabMonAgent.Dispose(Boolean disposing)
at TabMonService.TabMonServiceBootstrapper.Dispose(Boolean disposing)
at TabMonService.TabMonServiceBootstrapper.Dispose()
at Topshelf.Builders.ControlServiceBuilder1.ControlServiceHandle.Dispose()    at Topshelf.Runtime.Windows.WindowsServiceHost.Dispose(Boolean disposing)    at System.ComponentModel.Component.Dispose()    at System.ServiceProcess.ServiceBase.Run(ServiceBase[] services)    at System.ServiceProcess.ServiceBase.Run(ServiceBase service)    at Topshelf.Runtime.Windows.WindowsServiceHost.Run()    at Topshelf.HostFactory.Run(Action1 configureCallback)

I've had to back burner the TabMon implementation for now, with the intent that I would get back to it after the bug fix jmangue noted above was available.

Hi Jmangue,
I was curious wether the bug fixing made any progress?
best wishes

Hey all, sorry for the slow response on this one.  I've been researching this... it seems that the C# library for working with Perfmon requires that the target machine be on the same domain for remote polling.
The fix is not as easy as I initially thought, but I'll keep digging at it.
In the meantime, a workaround for this would be to ensure that TabMon is installed on a host on the same domain, and then specify the hostname in the TabMon config to be the simple computer name of the target machine.

Hey Jmangue -
I'm having the same issues it seems.  I've been trying to install TABMON and I'm getting the
"Error 5: Access is Denied" when starting the TabMon Service.  Has there been any updates on this issue?

I've also confirmed that all of my machines are installed on the same domain.  So that's not the issue here either.

This issue has been released by v1.2. Closing now.


Delete expired KeyPackage, rotate KeyPackages

We still need to rotate the KeyPackage if there haven't been any Welcome messages processed in the last X months, but I'm not sure the best way to implement that check?

For the CI issue, just run cargo fmt at the root of the repo and commit the change!

Looks great! How do we plan to delete the key material from the old key package on the client once the new package is uploaded? Is the idea to pipe an expiry into MLS for the old package at the time that the new one is created?

My idea was just to just leave all KeyPackages in storage until they expire, based on the thought that they might still be used.


Spread splitter not keeping aspect ratio

When using split and rotate, regardless whether upscaling is set to keep the aspect ratio, the resulting three images will all be stretched to the full resolution. I'm unsure if this is a bug in behavior or whether I should be asking for an added option to have the aspect ratio preserved here and the images instead be centered.

@Zaphain
Do you have an input example ?

https://imgur.com/a/XQ0QNIR
On closer look, it seems the rotated image is fine, it's just the split ones. Those are definitely stretched. The correct aspect ones were just done in imagemagick to show what I mean.

I could confirm that issue happen to me too.
I run KCC in webtoon mode, option Strech/upscale had been auto-checked, but the ratio of the result are not keep

@Zaphain
I'm unable to reproduce your problem.
With those settings:

I get the following, where orig.jpg is the original image
result.zip

...With those settings:

Okay, I see where the issue is, this doesn't happen when epub is chosen, it's only affecting the KFX output. https://imgur.com/a/uqDTyo7

OK, it must be here in the code.
KFX mode is stretching for split images
https://github.com/darodi/kcc/blob/88ffcefa6649e2d2a16812479691892c4a1ad186/kindlecomicconverter/image.py#L312-L313

I've tested with KFX Conversion Output Plugin without the stretch.
It works. I'll commit this and make a new release.
I must admit I've never used the KFX output before, only MOBI files.

@darodi
Could you please check this issues with these setting. The ratio is not keep


(maint) Pin Rake to 10.4, work around Rubocop change

Required for Ruby 1.8.7 testing.

CLA signed by all contributors.

👍


Ctrl-[ does not act as Escape in vim mode

Describe the bug
Somehow, I've trained myself to use Ctrl-[ as escape in vim and vim-likes (much closer to the home row than the actual Escape key). However i
Environment
{
"marimo": "0.1.76",
"OS": "Linux",
"OS Version": "6.5.0-5-amd64",
"Processor": "",
"Python Version": "3.12.1",
"Binaries": {
"Chrome": "--",
"Node": "v18.19.0"
},
"Requirements": {
"black": "23.9.1",
"click": "8.1.7",
"jedi": "0.19.1",
"pymdown-extensions": "10.7",
"tomlkit": "0.12.1",
"tornado": "6.3.3",
"typing_extensions": "4.8.0"
}
Code to reproduce
(none)

@inducer - seems like part of your issue got stripped

also, fwiw, I cannot produce this on Mac/Chrome, so likely an OS issue with Linux

seems like part of your issue got stripped

Thanks! Edited. Not sure where it went, I distinctly remember typing the rest of it... :shrug:

I can reproduce it on Linux/Chrome. I will look into it


chore: prepare v6.0.1

Description
This PR prepares the CHANGELOG and other files to release version v6.0.1.

Author Checklist
All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.
I have...

[ ] included the correct type prefix in the PR title
[ ] added ! to the type prefix if API or client breaking change
[ ] targeted the correct branch (see PR Targeting)
[ ] provided a link to the relevant issue or specification
[ ] followed the guidelines for building modules
[ ] included the necessary unit and integration tests
[ ] added a changelog entry to CHANGELOG.md
[ ] included comments for documenting Go code
[ ] updated the relevant documentation or specification
[ ] reviewed "Files changed" and left comments if necessary
[ ] confirmed all CI checks have passed

Reviewers Checklist
All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.
I have...

[ ] confirmed the correct type prefix in the PR title
[ ] confirmed ! in the type prefix if API or client breaking change
[ ] confirmed all author checklist items have been addressed
[ ] reviewed state machine logic
[ ] reviewed API design and naming
[ ] reviewed documentation is accurate
[ ] reviewed tests and test coverage
[ ] manually tested (if applicable)

I don't think we should release v6.0.1 already. The Stargate query fix is a small one and we should wait for other things first


Empty contextPath is replaced with "/", causing jsp page compilation error with tomcat 7

The short version
Gretty currently swaps out empty contextPath strings for "/". When using tomcat 7 (at least 7.0.65), that causes a very specific bug where JSP compilation will break on the root servlet. You'll get an error about negative time on all JSP's.
Doing this swap is the wrong thing to do for tomcat, and passing an empty string as the contextPath fixes the issue. I have a patch for this and I can submit a pull request if you want. However, you might not want to fix the issue the way I have (see long version).
The long version
According to the tomcat docs, the root context should be specified by an empty string (""). However, gretty currently fixes up contextPath strings that don't start with a "/" in WebappConfig.prepareToRun(), by adding a "/" prefix. The result is you can't specify the "" context path you need to.
The easy (and ignorable) symptom of this situation that tomcat spits on error along the lines of:
A context path must either be an empty string or start with a ''/'' and do not end with a ''/''...

However, the not-immediately-obvious symptom is that JSPs won't compile due to the "negative time" error I mentioned earlier.  Since this bug is very specific, I'm probably the only person to run into it.
--- Edit ---
The pull request fixes the issue by updating TomcatServerConfigurer to swap "/" to an empty string before setting the tomcat context name/path.

Great find - Running into the same issue.
I have been trying to figure out why farmRun was working on Jetty but not Tomcat.
Thanks

@ghar thanks! I figured I was the only person effected. Hopefully we can get the patch integrated soon.

@tkent If I fork your repo, is there a way for my build to pull from my fork on Github?

@ghar you can certainly use the patch right now.  Here's how you would do it.

Locally clone my repo
Checkout branch t192

If you only care about using it locally:

Publish the project to your local maven repository - ./gradlew publishToMavenLocal

Providing you use mavenLocal() in your project that needs the patch, that's it. Next time you build your project with a --refresh-dependencies you'll pickup the new version and it will just work.
If you want to share it amongst a team:
You'll need to have access to an artifact repository that you can publish to, and that same repository will need to be used in the projects that need the patch. Providing you have that, then...

Update the version property in gradle.properties to something that makes sense. We used 1.2.5-t192-SNAPSHOT
Modify libs/common.gradle so it publishes to the repo you have access to.

For us, since we don't happen use artifactory, we had to comment out the apply plugin: 'com.jfrog.**' lines, remove the bintrayandartifactory` closures, then add a publishing closure specific for our repo.
good luck!

Closing this issue since the changes in my pull request (#194) have been integrated into the master branch.
It looks like they will be available in this repository in gretty 1.4.0


Update README.md

Added how to add Raygun and Serilog packages

The code snippet now kind of looks like it's part of the NuGet explanation. I'd suggest either adding some more text above the code snippet to split it up a little - or pull the NuGet docs up into a different section with a different title.

Hows that?

That layout looks better to me, thanks. One other thought I had is that mentioning "command line" and "Program.Main" may be a bit specific. Instead, you could use the VS NuGet package manager, or set up the logger in Global.asax.Application_Start. Perhaps this could be written in a more generic way - let me know what you think about this.


Add ability to generate NuGet spec from the package manager console

Would be nice to have a quick way to move from a .nuspec-less NuGet.Packer project to creating a .nuspec file to customize the resulting package.
Would be nice to have a PowerShell command to do this from the NuGet Package Manager Console.

Or, it maybe a Visual Studio extension (ala #12) would be a good way to handle this, too!