Globus GCSv5 Endpoints need the data_access consent added

A NERSC user (@mzelinka) noticed an issue transferring data with zstash using globus on Perlmutter.
ERROR: ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 403, 'ConsentRequired', 'Missing required data_access consent', 'fD5Thunop')

This looks to be an error coming from newer GCSv5 collections, like the "NERSC Perlmutter" endpoint which require extra consents to be added in order to transfer data. The globus SDK has some documentation on how you can handle getting the data access consent for endpoints that require it.
NERSC is in the process of updating all our endpoints to the newest version of Globus (GCSv5) and all other Globus endpoints will need to updated before the older version GCSv4 is no longer supported in December 2023.
Full error logs:
INFO: Final Step of 3-legged OAuth2 Flows: Exchanging authorization code for token(s)
INFO: Fetching new token from Globus Auth
INFO: request done (success)
INFO: Setting up RefreshTokenAuthorizer with auth_client=[instance:140250644028176]
INFO: Setting up a RenewingAuthorizer. It will use an auth type of Bearer and can handle 401s.
INFO: RenewingAuthorizer will start by using access_token with hash "..."
INFO: Setting up RefreshTokenAuthorizer with auth_client=[instance:140250644028176]
INFO: Setting up a RenewingAuthorizer. It will use an auth type of Bearer and can handle 401s.
INFO: RenewingAuthorizer will start by using access_token with hash "..."
INFO: Creating client of type <class 'globus_sdk.services.transfer.client.TransferClient'> for service "transfer"
INFO: TransferClient.endpoint_autoactivate(6bdc7956-fc0f-4ad2-989c-7aa5ee643a79)
INFO: request done (success)
INFO: TransferClient.endpoint_autoactivate(9cd89cfd-6d04-11e5-ba46-22000b92c6ec)
INFO: request done (success)
INFO: TransferClient.operation_ls(9cd89cfd-6d04-11e5-ba46-22000b92c6ec, {'path': '/home/projects/e3sm/www/WaterCycle/E3SMv2/LR/v2.LR.amip_0101'})
INFO: request done (success)
INFO: Creating a new TransferData object
INFO: TransferClient.get_submission_id(None)
INFO: request done (success)
INFO: TransferData.DATA_TYPE = transfer
INFO: TransferData.DATA = []
INFO: TransferData.source_endpoint = 9cd89cfd-6d04-11e5-ba46-22000b92c6ec
INFO: TransferData.destination_endpoint = 6bdc7956-fc0f-4ad2-989c-7aa5ee643a79
INFO: TransferData.label = v2LRamip_0101 index
INFO: TransferData.submission_id = ...
INFO: TransferData.verify_checksum = True
INFO: TransferData.preserve_timestamp = True
INFO: TransferData.encrypt_data = False
INFO: TransferData.skip_source_errors = False
INFO: TransferData.fail_on_quota_errors = True
INFO: TransferData.delete_destination_extra = False
INFO: TransferData.notify_on_succeeded = True
INFO: TransferData.notify_on_failed = True
INFO: TransferData.notify_on_inactive = True
INFO: TransferClient.submit_transfer(...)
INFO: request done (success)
ERROR: ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 403, 'ConsentRequired', 'Missing required data_access consent', 'fD5Thunop')

Nick

It's not obvious to me from what I know about the issue yet, but is it possible user is trying to work with data on NERSC tape and NERSC scratch, but using globus?  I've not actually tried to use the new globus feature, but is it expected to work on the same machine (ie pm-to-pm)?

is it expected to work on the same machine (ie pm-to-pm)?

Yes, the testLsGlobus unit test runs pm-to-pm.

@tylern4 Does this have a work-around that does not require a code change? I.e., can consent be granted via the Globus website before running zstash, to avoid this issue? If not, the just-released version of zstash will not work for anyone using Globus... we would need to do a patch release of zstash with this bug fix.
(I've started trying to make the necessary code changes for a sustainable solution in #300).

I believe this code will give the needed to allow the zstash client_id the data access consent for a GCSv5 endpoint even without modifying your current code. It will print out the URL which you can authenticate with to give the client_id the data access consent.
import argparse
import globus_sdk

CLIENT = globus_sdk.NativeAppAuthClient(client_id="6c1629cf-446c-49e7-af95-323c6412397f",
                                        app_name="Zstash"
                                        )

def globus_flow(ep=""):
    scopes = f"urn:globus:auth:scope:transfer.api.globus.org:all"
    endpoint_scope = f"[ *https://auth.globus.org/scopes/{ep}/data_access ]"
    data_access_sopes = scopes + endpoint_scope
    CLIENT.oauth2_start_flow(refresh_tokens=True, requested_scopes=data_access_sopes)
    authorize_url = CLIENT.oauth2_get_authorize_url()
    print(f'Please go to this URL and login:\n\n{authorize_url}\n\n')

    get_input = getattr(__builtins__, 'raw_input', input)
    auth_code = get_input(
        'Please enter the code you get after login here: ')
    token_response = CLIENT.oauth2_exchange_code_for_tokens(auth_code)
    print(token_response)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("uuid", help="GCSv5 endpoint UUID")
    args = parser.parse_args()
    globus_flow(args.uuid)

@tylern4 Thanks, that does indeed work! That is very good as it means we don't need to rush a patch release of zstash then. That said, I'm assuming we'll want to incorporate this logic into the next release to streamline the process for users.
For reference, steps I followed:
# Copy the above to zstash_globus_consent.py
$ python zstash_globus_consent.py "6bdc7956-fc0f-4ad2-989c-7aa5ee643a79" # Perlmutter endpoint uuid
# Copy Link
# Log in to Globus
# Name the consent
# Copy the code and paste it to the terminal
$ source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
$ python -m unittest tests/test_globus.py
# Test passes

Relevant discussion post: #302

@tylern4 I'm now running into further difficulties.
When I run the test on the main branch, I now get a long error message beginning with ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 403, 'PermissionDenied', 'Error validating login to endpoint \'NERSC Perlmutter (6bdc7956-fc0f-4ad2-989c-7aa5ee643a79)\. It goes on to say None of your identities are from domains allowed by resource policies.
Furthermore, when I run the python zstash_globus_consent.py line, I don't see any consents populate at https://auth.globus.org/v2/web/consents.

Have you logged into Globus with your NERSC identity recently? The error None of your identities are from domains allowed by resource policies points to you being logged into Globus, but not logged into a NERSC identity.

I'm logged into the Globus web site. I used to enter my NERSC credentials for the Cori endpoint, which required activation. The Perlmutter endpoint doesn't seem to require activation though, so I'm not sure where I would enter those credentials now.

I tried just activating a NERSC endpoint I didn't actually need (in order to use my NERSC credentials), but I still run into the same "Permission denied" error.

@tylern4 @lukaszlacinski Yeah, still running into this issue -- how would I "log into a NERSC identity"? I'm logged into Globus, I have NERSC endpoints activated, I'm not sure what else I could do.

It's related to this error in Globus. At NERSC we require you to log in with your @nersc.gov domain to access our endpoints, if you can login and see your data in the NERSC Perlmutter collection you should also be able to use it from your tool when you're logged in. Maybe you need to refresh your token loaded into zstash?

see your data in the NERSC Perlmutter collection

Oh interesting; while it doesn't require endpoint activation, it does require me to enter my credentials to review the contents in the file manager.
I'm now able to run with the latest Unified once again, so I can get back to #304. Thanks @tylern4!

@tylern4 Following https://github.com/E3SM-Project/zstash/discussions/302#discussioncomment-7139158, if I go to https://auth.globus.org/v2/web/consents (either by direct link or by following "Settings" > "Consents" > "Manage Your Consents"), I only see a couple consents, both of which were granted 2 years ago. There is nothing specific to the Perlmutter endpoint.
I went to the Perlmutter endpoint (https://app.globus.org/file-manager/collections/6bdc7956-fc0f-4ad2-989c-7aa5ee643a79/overview) > "Manage Consent", where I did see relevant consents. However, I deleted them and tried re-running zstash with Globus and it worked fine. I would have expected to need to re-establish the consents. Am I missing something? Thanks.

@tylern4 Expanding on this, I did a few manual tests (as opposed to using test_globus.py):
On Chrysalis:
# Create something to archive
$ emacs setup.sh
mkdir zstash_demo
mkdir zstash_demo/empty_dir
mkdir zstash_demo/dir
echo 'file0 stuff' > zstash_demo/file0.txt
echo '' > zstash_demo/file_empty.txt
echo 'file1 stuff' > zstash_demo/dir/file1.txt
$ chmod 755 setup.sh
$ ./setup.sh
$ source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
$ zstash create --hpss=globus://nersc/~/n298_chrysalis zstash_demo

# ERROR: The 61f9954c-a4fa-11ea-8f07-0a21f750d19b endpoint is not activated or the current activation expires soon. Please go to https://app.globus.org/file-manager/collections/61f9954c-a4fa-11ea-8f07-0a21f750d19b and (re)activate the endpoint.
# => Activate lcrc#dtn_bebop endpoint via web interface

$ zstash create --hpss=globus://nersc/~/n298_chrysalis zstash_demo

# ERROR: ('POST', 'https://transfer.api.globus.org/v0.10/transfer', 'Bearer', 403, 'ConsentRequired', 'Missing required data_access consent', 'ghHoWXlRW')
# => Run:
$ python zstash_globus_consent.py "61f9954c-a4fa-11ea-8f07-0a21f750d19b" # lcrc#dtn_bebop endpoint

# Error on Globus website:
# client_id=6c1629cf-446c-49e7-af95-323c6412397f requested unknown scopes: ['https://auth.globus.org/scopes/61f9954c-a4fa-11ea-8f07-0a21f750d19b/data_access']
# => Run:
$ python zstash_globus_consent.py "9cd89cfd-6d04-11e5-ba46-22000b92c6ec" # NERSC HPSS endpoint
# => The consent script needs the destination endpoint only, not the source endpoint.

$ zstash create --hpss=globus://nersc/~/n298_chrysalis zstash_demo

# Now, on Perlmutter:
$ hsi
$ cd n298_chrysalis
$ ls
# 000000.tar   index.db
# Success

Then, I ran on Perlmutter:
# Same setup script as above
$ source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
$ zstash create --hpss=globus://nersc/~/n298_pm zstash_demo

$ hsi
$ cd n298_pm
$ ls
# 000000.tar   index.db
# Success

# Disable consents to test consent activation
# Go to NERSC HPSS on web interface
# https://app.globus.org/file-manager/collections/9cd89cfd-6d04-11e5-ba46-22000b92c6ec/overview
# > Manage Consent
# No consents found
# Go to settings page on web interface
# https://app.globus.org/settings/identities
# > Consents
# > Manage Your Consents
# All consents are from years ago

So, clearly I'm not disabling consents correctly. That's crucial to testing any consent automation.

It's not the endpoints you need to delete consents from to test, it's the consents you've given to your tool that you would need to delete.
You've given the client_id the name Globus Endpoint Performance Monitoring so in the Globus Manage Consents page you'll want to find that row, which should say [Native App] next to it, open the dropdown and remove all the consents you have given to it.


More pure4: favorites/git

More pure actions in favorites / git
Also fixes a bug in git where if you choose 'new team' in create team repo it sends you to the teams page with the new team dialog

@keybase/react-hackers

@MarcoPolo is it ok for me to merge this


[RFC] Recipe PEFT setup refactor

Just some quick thoughts I'm dumping here for feedback. I feel as though our LoRA recipes are laden with PEFT-related utils which don't necessarily need to be in the recipes. This bugged me as I've been working on a LoRA PPO recipe (which requires PEFT setup for two models). This would look horrendous. I'd like to propose the small refactor I used.
This is the current LoRA setup:
    def _setup_model(def _setup_model(
        self,
        cfg_model: DictConfig,
        enable_activation_checkpointing: bool,
        compile_model: bool,
        base_model_state_dict: Dict[str, Any],
        lora_weights_state_dict: Optional[Dict[str, Any]] = None,
    ) -> nn.Module:
        with training.set_default_dtype(self._dtype), self._device:
            model = config.instantiate(cfg_model)

        self._lora_rank = cfg_model.lora_rank
        self._lora_alpha = cfg_model.lora_alpha
        self._lora_attn_modules = list(cfg_model.lora_attn_modules)
        self._apply_lora_to_mlp = cfg_model.apply_lora_to_mlp
        self._apply_lora_to_output = getattr(cfg_model, "apply_lora_to_output", False)
        self.adapter_params = get_adapter_params(model)
        self._is_dora = any(["magnitude" in k for k in self.adapter_params.keys()])
        set_trainable_params(model, self.adapter_params)

        if compile_model:
            log.info("Compiling model layers with torch.compile...")
            backend = os.environ.get("TORCH_COMPILE_BACKEND", "inductor")
            for m in reversed(list(model.modules())):
                if isinstance(m, modules.transformer.TransformerSelfAttentionLayer):
                    m.compile(backend=backend)

        if enable_activation_checkpointing:
            utils.set_activation_checkpointing(
                model, auto_wrap_policy={modules.TransformerSelfAttentionLayer}
            )

        base_missing, base_unexpected = model.load_state_dict(
            base_model_state_dict, strict=False
        )
        # This is for any adapters that need to be initialized after base weights
        # have been loaded (e.g. DoRA).
        if self._is_dora:
            load_dora_magnitudes(model)
        if lora_weights_state_dict:
            lora_missing, lora_unexpected = model.load_state_dict(
                lora_weights_state_dict, strict=False
            )
        else:
            lora_missing, lora_unexpected = None, None
        validate_missing_and_unexpected_for_lora(
            lora_attn_modules=self._lora_attn_modules,
            apply_lora_to_mlp=self._apply_lora_to_mlp,
            apply_lora_to_output=self._apply_lora_to_output,
            base_missing=base_missing,
            base_unexpected=base_unexpected,
            lora_missing=lora_missing,
            lora_unexpected=lora_unexpected,
        )
        # Validate model adapter params were loaded in with the expected dtype
        # TODO (rohan-varma): Further validation to ensure the appropriate base params
        # are NF4 vs bf16 based on the quantization config.
        training.validate_expected_param_dtype(
            self.adapter_params.items(), dtype=self._dtype
        )

        log.info(f"Model is initialized with precision {self._dtype}.")

        if self._device.type == "cuda":
            memory_stats = utils.get_memory_stats(device=self._device)
            utils.log_memory_stats(memory_stats)
        return model

It's pretty chonky. Verbose. This is more along the lines of what I'd like it to look like:
    def _setup_model(
        self,
        cfg_model: DictConfig,
        enable_activation_checkpointing: bool,
        compile_model: bool,
        base_model_state_dict: Dict[str, Any],
        lora_weights_state_dict: Optional[Dict[str, Any]] = None,
    ) -> nn.Module:

        with training.set_default_dtype(self._dtype), self._device:
            model = config.instantiate(cfg_model)

        # from peft/_utils.py - ModelPEFTConfig is a NamedTuple
        self._peft_config: ModelPEFTConfig = setup_model_peft_config(model, cfg_model)

        if compile_model:
            log.info("Compiling model layers with torch.compile...")
            backend = os.environ.get("TORCH_COMPILE_BACKEND", "inductor")
            for m in reversed(list(model.modules())):
                if isinstance(m, modules.transformer.TransformerSelfAttentionLayer):
                    m.compile(backend=backend)

        if enable_activation_checkpointing:
            utils.set_activation_checkpointing(
                model, auto_wrap_policy={modules.TransformerSelfAttentionLayer}
            )

        # from peft/_utils.py
        validate_load_model_peft_checkpoint(
            model,
            base_model_state_dict,
            self._peft_config,
            lora_weights_state_dict=lora_weights_state_dict,
        )

        training.validate_expected_param_dtype(
            self.adapter_params.items(), dtype=self._dtype
        )

        log.info(f"Model is initialized with precision {self._dtype}.")

        if self._device.type == "cuda":
            memory_stats = utils.get_memory_stats(device=self._device)
            utils.log_memory_stats(memory_stats)
        return model

Concretely, we keep track of the PEFT-related variables needed later for saving checkpoints in:
class ModelPEFTConfig(NamedTuple):
    lora_rank: int
    lora_alpha: float
    lora_attn_modules: List[LORA_ATTN_MODULES]
    apply_lora_to_mlp: bool
    apply_lora_to_output: bool
    adapter_params: Dict[str, nn.Parameter]
    is_dora: bool

    def to_checkpoint_adapter_config(self):
        return {
            "r": self.lora_rank,
            "lora_alpha": self.lora_alpha,
            "target_modules": get_lora_module_names(
                self.lora_attn_modules,
                self.apply_lora_to_mlp,
                self.apply_lora_to_output,
            ),
            "peft_type": "LORA",
        }

Then, validate_load_model_peft_checkpoint then takes care of validating and loading the base model + adapter weights. The reason for to_checkpoint_adapter_config is so you can do
        ckpt_dict.update(
            {training.ADAPTER_CONFIG: self._peft_config.to_checkpoint_adapter_config()}
        )

in save_checkpoint.  You'd also use self._peft_config to grab all the other relevant variables for saving checkpoints.
cc @pbontrager who has previously expressed interest in moving utilities out of recipes.

I didnt look deeply into it, but at first glance, it makes sense to me. The issue is the the principle that torchtune wants to avoid any sort of nested configs, e.g. ModelPEFTConfig. So if feels a bit hard to sell it. But if we did go down that route, it could probably simplify issues with LoRA vs DoRA vs MoRA etc.

Thanks for pointing that out. I've thought of a simpler solution for my use-case I can try out here rather than messing with the config system.


Orientation Events doesn't work when rotate device from "LANDSCAPE" to "PORTRAIT"

Hi I upgrade to v3.1.3 and I found the Orientation Events seems broke when rotate device from "LANDSCAPE" to "PORTRAIT". The "LANDSCAPE" event won't be triggered. Now I have to roll back to v3.1.0 again...

Which OS?

@stoneman1 iOS 11

Ok will test soon and try to fix.

I seem to have a similar issue where the Orientation event won't be triggered, it is always at portrait and doesn't change to Landscape. I have checked everywhere if I have locked the orientation, but it seems like I haven't and it is acting as if the orientation is locked. Also The same code works perfectly on Android.
P.S. Before I added react-native-orientation, my app would automatically readjust to the orientation.


[RLMAccessor] Improve the exception message when trying to set a property with an object of the incorrect type.

Closes https://github.com/realm/realm-cocoa/issues/1369.
\c @alazier @tgoyne

:+1: once tests pass.


respond to https://github.com/sparcopen/opencon/issues/4 with changes

I've hit some, but not all of this.

have added to do's and some comments here

okay! I've done everything fast here. Other stuff will probably need more than I'll be able to do now.

okay, done a few slower things there and updated people.

moving to the next issue, everything else I think will take more time than we have today


Add tutorials on code generation and RK order conditions to docs.

In this PR I added two tutorials, based on user requests:

Generation of C code using symbolics.jl.  It would be nice if we could also use Sympy's code generation utilities, but I can't find a way to access those from SymPy.jl.  I've raised an issue over there.
Generation of RK order conditions for a RK method with unspecified coefficients.

For both of these, I've also included a Jupyter notebook source.
I don't know how to add these to the table of contents of the docs, or whether that will happen automatically.

For the order conditions tutorial, it would be nice if we could also have output where the indices of A and b appear properly as subscripts in the output (like they do when you print out the matrices themselves) but I couldn't find a way to do that.

Thanks!  I followed all your suggestions.  Hopefully this is ready now.

The docs are failing to build, and I get the same error when building them locally, but I don't understand why.   The output from the build is not very clear to me.  I think the key is these lines:
ERROR: LoadError: UndefRefError: access to undefined reference
Stacktrace:
  [1] getindex
    @ ./array.jl:861 [inlined]
  [2] (::SymPy.var"#2#5"{Vector{SymPy.Sym}})(i::Int64)

It seems clear that this indicates an issue with RK_order_conditions.md, which has these lines:
A = Array{Sym,2}(undef, s, s)
b = Array{Sym,1}(undef, s)
c = Array{Sym,1}(undef, s);

However, all the values of these arrays are later defined before they are used.  I executed all the code in that file in a julia prompt and it ran just fine.  It also (of course) runs fine in the corresponding Jupyter notebook.  @ranocha do you have an idea what could be causing this error?


[FEATURE REQUEST] Helm part of local usage

Is your feature request related to a problem? Please describe.
In #1638 it was decided not to wrap kubectl to execute locally and example provided how to change configuration in #1639.
This task requires investigation what needs to be changed for helm part of that work.
Describe the solution you'd like
Investigate and apply what needs to be changed for helm part of execution in the local container.
Describe alternatives you've considered
No.
Additional context
See example.

Using helm locally to deploy charts to a K8s cluster can be done with the changes already made for #1616:

Local kube.conf which points to localhost:6443
Setting of KUBECONF env variable to point to that local kube.conf
SSH tunnel passing through port 6443 to our K8s master

Only thing missing is adding our own Helm repository which we can do by using the same SSH tunnel trick we use for local kubectl:

SSH tunnel passing through port 80 to our repository machine
Add the repository using localhost address instead of the normal IP
deploy charts like normal.

It has been tested together with the task [FEATURE REQUEST] update network security rule to open port 6443 #1616.


Fix the issue of the report on mac Bug#858

Fix the issue of the report on mac
There were actually two things :-
1- First of all it was CD'ing to generate_allure_report.sh which is No such file or directory so I changed the '' to ""

After the Fix

2- The other thing and the one that caused the issue is $parent_path this variable was read as String so it couldn't be appended the variable value well

And here is the generate_allure_report.sh After generation

I'm surprised no one cried out that it's also broken on linux 😂
Thanks a lot for the clean fix 👌


Request: Notify User, which word is being blocked.

One thing, I feel could improve this plugin, is when a word is blocked, have it relay that to the user.
Often there are a number of false positives, which can make it difficult to determine which word is tripping it up. This small change could help limit the frustration of players having to retype a message multiple times trying to figure out which word it's unhappy with.

Thanks for your request, has been added to the latest version as an option to the config (off by default) 👍


Segpipe cmd

echo hi | cat <"./minishell_tester/test_files/infile"

doesn't happen in the workspace enviroment, only personal wsl space


Request to add a site: viz.com/shonenjump

Request to add a site

Description
Data

Url to add
https://www.viz.com/shonenjump/chapters

I tried the 0.x executable, but it didn't download and closed the terminal immediately. I am subscribed to viz's shonen jump, I would also like to request a feature to login with my account when downloading. Thanks.

OK. I will try to do it.
Version 0.x is not supported now. The executable file is working, but it needs to be called from the command line. It does not have a GUI.

So. About viz
It's complicated --
The fact is that I need an account to see what it looks like on behalf of a registered user (or at least page dumps)
In addition, unfortunately, the site is not available in my country (I use vpn in Italy and the United States to fix it, but it does not always help --)
May I ask you to send an archive with pages?
Looking for the following data:

A page with a list of chapters (preferably with a manga, where there is a paginator, if it is used on this site)
Page with pictures

You can save simply through CTRL + S

I hope you understand me. Sorry for my English :-(

There are free access to the latest few chapters. Just for example:

https://www.viz.com/shonenjump/chapters/dr-stone
https://www.viz.com/shonenjump/dr-stone-chapter-106/chapter/18224?action=read

So the images are obfuscated for copyright protection, like this.

images are obfuscated

its not problem
Thx for snapshots. I'll do this a little later.

For several reasons, work on the provider for viz is delayed. Sorry

First step result. yay!

And ... Done!
It really works!
After about 4-5 hours there will be a release.

Phew It was complicated )
Done!
Unfortunately, it works ... strange
The number of images will temporarily show 250
But do not worry. Only those that are in the manga should be archived.
Please check it.
It was really difficult for me to work with this site, because it is not accessible from Russia :-(
Anyway. If he works for you, I will be glad
If you have any difficulties, feel free to write
If it works, please wait a little longer. I will try to correct this oddity with the number of pages.

Fails gracefully out of USA and Japan. (Works as intended)
...
Try to get free chapters...
Warning! This site worked from USA and Japan! Check your location

@gromenauer yes, that's right.
Unfortunately, I can check only Russia, Italy and the USA (maybe it works in some other countries. Not sure)

Phew It was complicated )
Done!
Unfortunately, it works ... strange
The number of images will temporarily show 250
But do not worry. Only those that are in the manga should be archived.
Please check it.
It was really difficult for me to work with this site, because it is not accessible from Russia :-(
Anyway. If he works for you, I will be glad
If you have any difficulties, feel free to write
If it works, please wait a little longer. I will try to correct this oddity with the number of pages.
P.S.
If you leave the login or password field blank, the provider will be started in guest mode

I upgraded to version <IP_ADDRESS>, then tried

(base) C:\Users\roba7\workspace\manga>manga-py https://www.viz.com/shonenjump/dr-stone-chapter-114/chapter/18621?action=read
Provider not found
Please check the domain in the table: https://yuru-yuri.github.io/manga-dl/
Make sure that the URL is correct
Trace:
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py_init_.py", line 43, in init_cli
cli_mode.start()
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py\cli_init.py", line 40, in start
raise e
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py\cli_init_.py", line 34, in start
info=self._info,
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py\parser.py", line 35, in init_provider
raise AttributeError('Provider not found')

and somehow your provider list link is dead https://yuru-yuri.github.io/manga-dl/
Thanks for all the work, and keep it up.

Ok I figured out. The command should be like "manga-py https://www.viz.com/shonenjump/chapters/dr-stone" So it turns out that it works and the image downloaded aren't obfuscated, but even with my paid membership, I'm getting only the free chapters, that are first three and last three chapters.

@mugenZebra thanks for the information
Can you send a page with a list of chapters?
(CTRL + S in your browser)
Maybe I get the wrong list of chapters.

Hey. If you are still interested in this issue, please send the archive with the page here (github allows you to attach archives to the answer).
Thank.

@mugenZebra thanks for the information
Can you send a page with a list of chapters?
(CTRL + S in your browser)
Maybe I get the wrong list of chapters.
P.S.
About the link. I changed the name of the repository a bit so that there was less confusion
See here: https://yuru-yuri.github.io/manga-py
P.P.S.
Oh. Thank. I forgot to change the hint link for manga-py

sorry for the late reply. Here's another snapshot saved with Firefox. Thanks for working on it.
https://file.io/fRKvro

https://file.io/fRKvro
404 Not Found

shonenJumpviz.zip
Great, didn't know that.

Oh!  I understood my mistake. Thank!

P.S. in the directory from which manga-py will be launched, the debug-viz directory will be created.
Please send her too.
To get the log to the file, you can use the command >
eg:
manga-py example.org > file.log
Thanks.

Hi sorry for the late reply, didn't see it until now. Here's the log.
log.log
And the viz_debug file:
chapter_5558_page_0.txt
it's supposed to be .html but github doesn't let me upload it so I changed the extension.

Please make an archive for the debug-viz directory
Or was there only 1 file of zero size?

Hi,
I tried, here's the generated files.
Manga.zip
I also got this error message in the commandline:

2019-09-10 21:20:30.211 | ERROR    | manga_py.cli:start:41 - An error has been caught in function 'start', process 'MainProcess' (7616), thread 'MainThread' (7532):
Traceback (most recent call last):
File "c:\users\roba7\anaconda3\lib\runpy.py", line 193, in run_module_as_main
"main", mod_spec)
└ ModuleSpec(name='main', loader=<zipimporter object "C:\Users\roba7\Anaconda3\Scripts\manga-py.exe">, origin='C:\Users\roba...
File "c:\users\roba7\anaconda3\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
│    │     └ {'name': 'main', 'doc': None, 'package': '', 'loader': <zipimporter object "C:\Users\roba7\Anaconda3\Scripts...
│    └ <code object  at 0x0000025C42F5E930, file "C:\Users\roba7\Anaconda3\Scripts\manga-py.exe_main.py", line 2>
└
File "C:\Users\roba7\Anaconda3\Scripts\manga-py.exe_main.py", line 9, in
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py_init_.py", line 96, in main
code, _info = _run_util(args)
│         └ ArgumentParser(prog='manga-py', usage=None, description=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler...
└ <function run_util at 0x0000025C436AF510>
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py_init.py", line 55, in _run_util
code = _init_cli(args, _info)
│         │     └ <manga_py.info.Info object at 0x0000025C43699EF0>
│         └ ArgumentParser(prog='manga-py', usage=None, description=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler...
└ <function init_cli at 0x0000025C436AF488>
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py_init.py", line 43, in _init_cli
cli_mode.start()
│        └ <bound method Cli.start of <manga_py.cli.Cli object at 0x0000025C43699D68>>
└ <manga_py.cli.Cli object at 0x0000025C43699D68>

File "c:\users\roba7\anaconda3\lib\site-packages\manga_py\cli_init_.py", line 41, in start
self.parser.start()
│    │      └ <bound method Parser.start of <manga_py.parser.Parser object at 0x0000025C43699E48>>
│    └ <manga_py.parser.Parser object at 0x0000025C43699E48>
└ <manga_py.cli.Cli object at 0x0000025C43699D68>
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py\parser.py", line 45, in start
self.provider.process(self.params['url'], self.params)
│    │        │       │    │              │    └ {'url': 'https://www.viz.com/shonenjump/chapters/dr-stone', 'name': '', 'destination': 'Manga', 'no_progress': True, 'not_change...
│    │        │       │    │              └ <manga_py.parser.Parser object at 0x0000025C43699E48>
│    │        │       │    └ {'url': 'https://www.viz.com/shonenjump/chapters/dr-stone', 'name': '', 'destination': 'Manga', 'no_progress': True, 'not_change...
│    │        │       └ <manga_py.parser.Parser object at 0x0000025C43699E48>
│    │        └ <bound method Provider.process of <manga_py.providers.viz_com.VizCom object at 0x0000025C43699F28>>
│    └ <manga_py.providers.viz_com.VizCom object at 0x0000025C43699F28>
└ <manga_py.parser.Parser object at 0x0000025C43699E48>
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py\provider.py", line 81, in process
self.prepare_cookies()
│    └ <bound method VizCom.prepare_cookies of <manga_py.providers.viz_com.VizCom object at 0x0000025C43699F28>>
└ <manga_py.providers.viz_com.VizCom object at 0x0000025C43699F28>
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py\providers\viz_com.py", line 93, in prepare_cookies
self.auth()
│    └ <bound method VizCom.auth of <manga_py.providers.viz_com.VizCom object at 0x0000025C43699F28>>
└ <manga_py.providers.viz_com.VizCom object at 0x0000025C43699F28>
File "c:\users\roba7\anaconda3\lib\site-packages\manga_py\providers\viz_com.py", line 127, in auth
self.__cookies['remember_token'] = remember['trust_user_id_token_web']
│                                  └ {'ok': 1}
└ <manga_py.providers.viz_com.VizCom object at 0x0000025C43699F28>

KeyError: 'trust_user_id_token_web'

21 day ...
Ohh...
The site changed the storage of cookies.
Please do the same with the version 1.8.5-rc.1

@mugenZebra
There is no connection with you.
If your problem is resolved, please close issue.
If not, I'm still waiting for an answer.

Doesn't work anymore.
moot@7c09751204d7:~/manga-py# python3 manga.py https://www.viz.com/shonenjump/chainsaw-man-chapter-1/chapter/17489
Provider not found
Please check if your inputed domain is supported by manga-py:
- https://manga-py.com/manga-py/#resources-list
- https://manga-py.github.io/manga-py/#resources-list (alternative)
- https://yuru-yuri.github.io/manga-py/ (deprecated)
Make sure that your inputed URL is correct

Trace:
  File "/root/manga-py/manga_py/__init__.py", line 43, in _init_cli
    cli_mode.start()
  File "/root/manga-py/manga_py/cli/__init__.py", line 43, in start
    raise e
  File "/root/manga-py/manga_py/cli/__init__.py", line 34, in start
    info=self._info,
  File "/root/manga-py/manga_py/parser.py", line 35, in init_provider
    raise AttributeError('Provider not found')

Hey. I will try to check it
Unfortunately, from my country there is no access to this section.


Err: Max Connections

I'm running into a max connections error when running on Heroku: ERR max number of clients reached
When I do a client list in Redis I see a ton of connections that look like this:
id=28 addr=<IP_ADDRESS>:59295 fd=33 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=114 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=29 addr=<IP_ADDRESS>:59296 fd=34 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=450 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=30 addr=<IP_ADDRESS>:59302 fd=35 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=477 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=31 addr=<IP_ADDRESS>:59298 fd=36 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=188 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=32 addr=<IP_ADDRESS>:59297 fd=37 name= age=14 idle=14 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=188 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop`

How many connections does Kue.js normally need to create?
Are the number of connections directly related to how many job types I'm creating/processing?
If i have 10 different job types, does that mean I need 10 different connections to listen for each type?

I have the same problem. I see a new connection opened for each job type. I think this is not manageable if the application needs many job types, especially if using redis solutions from the cloud (e.g. redis labs has maximum 30 connections for free plan).
A work around may be to implement the redis createClientFactory option to return always one client. However, I am not sure if having a shared client may create problems.
Any help on this?

It would be great to document how many redis connections are used. I have an app with multiple workers, and with 6 workers I somehow have 39 connections open to redis. I only have one job type, but ~3k jobs of that type in the queue.


ipython-sql unable to query Pandas DF

Hello,
Thank you again for the help you have already provided! DuckDB is working for any tables within DuckDB now in ipython-sql.
My current issue is that DuckDB's ability to query directly on top of a Pandas DataFrame does not appear to be working. Ideally, this query would not return a missing table error. I'm seeing this issue when using duckdb_engine directly as well as when using ipython-sql.
I included a working example using the direct DuckDB connection object below the traceback.
Thank you again!
-Alex
test_df2 = pd.DataFrame.from_dict({"i":[1, 2, 3, 4], "j":["one", "two", "three", "four"]})
engine = sqlalchemy.create_engine('duckdb:///a_test_duckdb.db')
df = pd.read_sql('select * from test_df2', engine)
df

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-57-da30afad1d72> in <module>
      1 test_df2 = pd.DataFrame.from_dict({"i":[1, 2, 3, 4], "j":["one", "two", "three", "four"]})
      2 engine = sqlalchemy.create_engine('duckdb:///a_test_duckdb.db')
----> 3 df = pd.read_sql('select * from test_df2', engine)
      4 df

C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\sql.py in read_sql(sql, con, index_col, coerce_float, params, parse_dates, columns, chunksize)
    525             coerce_float=coerce_float,
    526             parse_dates=parse_dates,
--> 527             chunksize=chunksize,
    528         )
    529

C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\sql.py in read_query(self, sql, index_col, coerce_float, parse_dates, params, chunksize)
   1306         args = _convert_params(sql, params)
   1307
-> 1308         result = self.execute(*args)
   1309         columns = result.keys()
   1310

C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\sql.py in execute(self, *args, **kwargs)
   1174     def execute(self, *args, **kwargs):
   1175         """Simple passthrough to SQLAlchemy connectable"""
-> 1176         return self.connectable.execution_options().execute(*args, **kwargs)
   1177
   1178     def read_table(

<string> in execute(self, statement, *multiparams, **params)

C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\util\deprecations.py in warned(fn, *args, **kwargs)
    388         if not skip_warning:
    389             _warn_with_version(message, version, wtype, stacklevel=3)
--> 390         return fn(*args, **kwargs)
    391
    392     doc = func.__doc__ is not None and func.__doc__ or ""

C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py in execute(self, statement, *multiparams, **params)
   3035         """
   3036         connection = self.connect(close_with_result=True)
-> 3037         return connection.execute(statement, *multiparams, **params)
   3038
   3039     @util.deprecated_20(

C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py in execute(self, statement, *multiparams, **params)
   1188                 params,
   1189                 _EMPTY_EXECUTION_OPTS,
-> 1190                 future=False,
   1191             )
   1192

C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py in _exec_driver_sql(self, statement, multiparams, params, execution_options, future)
   1489             execution_options,
   1490             statement,
-> 1491             distilled_parameters,
   1492         )
   1493

C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py in _execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1747         except BaseException as e:
   1748             self._handle_dbapi_exception(
-> 1749                 e, statement, parameters, cursor, context
   1750             )
   1751

C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py in _handle_dbapi_exception(self, e, statement, parameters, cursor, context)
   1931                 )
   1932             else:
-> 1933                 util.raise_(exc_info[1], with_traceback=exc_info[2])
   1934
   1935         finally:

C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\util\compat.py in raise_(***failed resolving arguments***)
    209
    210         try:
--> 211             raise exception
    212         finally:
    213             # credit to

C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\engine\base.py in _execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1704                 if not evt_handled:
   1705                     self.dialect.do_execute(
-> 1706                         cursor, statement, parameters, context
   1707                     )
   1708

C:\ProgramData\Anaconda3\lib\site-packages\duckdb_engine\__init__.py in do_execute(self, cursor, statement, parameters, context)
    132         context: PGExecutionContext,
    133     ) -> None:
--> 134         cursor.execute(statement, parameters, context)
    135
    136     def do_executemany(

C:\ProgramData\Anaconda3\lib\site-packages\duckdb_engine\__init__.py in execute(self, statement, parameters, context)
     89                 return
     90             else:
---> 91                 raise e
     92
     93

C:\ProgramData\Anaconda3\lib\site-packages\duckdb_engine\__init__.py in execute(self, statement, parameters, context)
     79                 self.c.execute(statement)
     80             else:
---> 81                 self.c.execute(statement, parameters)
     82         except RuntimeError as e:
     83             if e.args[0].startswith("Not implemented Error"):

RuntimeError: Catalog Error: Table with name test_df2 does not exist!
Did you mean "sqlite_schema"?
LINE 1: select * from test_df2

A working example using the DuckDB connection directly:
test_df3 = pd.DataFrame.from_dict({"i":[1, 2, 3, 4], "j":["one", "two", "three", "four"]})
try:
    conn2 = duckdb.connect(database='my_example_duckdb3.db', read_only=False)
    query_output = conn2.execute('select * from test_df3').fetchdf()
finally:
    conn2.close()
print(query_output)

Considering you're not registering the dataframe in either code snippet, I assume the second on disk database actually just has a table with that name?

The DuckDB folks added a feature that looks into Python's local variables for all DataFrames and they can now be queried without needing to register them. Here is their blog post on it!

Ah okay, that's a bit of a hacky feature - I'm not sure if it would even be possible for that to be supported by this library, considering how they implemented it.
For reference, the implementation is here: https://github.com/duckdb/duckdb/blob/9c0a350a7571272004dbc8d224689bb1aaa18963/tools/pythonpkg/src/pyconnection.cpp#L401-L431

Ah, ok. Any advice for how I could manually register a DF with DuckDB engine then?
Thanks!

I've just added (somewhat experimental) support in 0.1.8-rc.4
eng = create_engine("duckdb:///:memory:")
eng.execute("register", "dataframe_name", pd.DataFrame(...))

eng.execute("select * from dataframe_name")

Thank you! I will test that out shortly!

That works! Thanks!
In case anyone else is experimenting with ipython-sql, I've got a question out to those folks about how to access the underlying SQLAlchemy engine or pass in a pre-created one. That's the last missing puzzle piece for DuckDB SQL on DF's with friendly syntax highlighting in Jupyter!

For my own later reference, that issue is https://github.com/catherinedevlin/ipython-sql/issues/199

This is now solved! The DuckDB folks added support for this feature when using iPython.
PR that solves the issue


Subnetwork should support importing existing subnetwork resource

What problem are you facing?

When I create a new Network type in GCP, it uses the existing network without any issues. However, creating a new Subnetwork fails if that subnetwork exists.:
  apiVersion: compute.gcp.crossplane.io/v1beta1
  kind: Subnetwork
  ...
  spec:
    forProvider:
      ipCidrRange: <IP_ADDRESS>/12
      ...
    providerRef:
      name: gcp-provider
    reclaimPolicy: Retain
  status:
    atProvider: {}
    conditions:
    - lastTransitionTime: "2020-03-05T22:33:23Z"
      reason: Successfully resolved resource references to other resources
      status: "True"
      type: ReferencesResolved
    - lastTransitionTime: "2020-03-05T22:33:23Z"
      reason: Resource is being created
      status: "False"
      type: Ready
    - lastTransitionTime: "2020-03-05T22:33:23Z"
      message: 'create failed: creation of GCP Subnetwork resource has failed: googleapi:
        Error 400: Invalid value for field ''resource.ipCidrRange'': ''<IP_ADDRESS>/12''.
        Invalid IPCidrRange: <IP_ADDRESS>/12 conflicts with existing subnetwork ''basic-tier-subnetwork''
        in region ''us-central1''., invalid'
      reason: Encountered an error during resource reconciliation
      status: "False"
      type: Synced

How could Crossplane help solve your problem?

Would it be possible to make the subnetwork use the existing resource? This is handy when multiple resources are using the same subnetwork, but are being created in different settings and have a Retain policy for reclaiming.

This should be fixed by external-name annotation.


I'm not able to generate profiles other than default - stuck at 19/XXX

Hi !
I'm not able to generate profiles other than default ... the process stops here (see below) and nothing happens then ...
The GPU remains under heavy load as you can see below.
In this example I was trying to generate 768*768 static for cyberrealistic v33 but it happens with other models too.

Thanks in advance for a suggestion for a possible solution :-)
Paolo

Hey! How long did you wait for the engine generation? Depending on the configuration, it might take longer.

Hey! How long did you wait for the engine generation? Depending on the configuration, it might take longer.

Yes I waited for quite a long period, and it stops at 19/472.
I'm expecting to complete the task in the same amount of time of default profile.
In the past I managed to create a profile for 768 in a reasonable amount of time, but it's not the case anymore.
:-(

Does it work with SD1.5?

I'm only trying with 1.5 models

I have almost the exact same thing happening.

Closing as this seems to have been resolved. Sorry for the inconvenience, but sometimes TensorRT can be a bit funky when encountering unknown models.


⚠ Bofrost.lu has degraded performance

In f8b21ef, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1184 ms

Resolved: Bofrost.lu performance has improved in e45d2fe after 6 minutes.


apm upgrade

:boom: This brings in a prerelease version of apm based on atom/apm#796 to see how horribly it breaks everything. :boom:

It seems to be working... :pray:

@smashwilson I think we need to test some (major) community packages that use native node modules, to verify that they still compile (and run) with the upgrade.

@thomasjo Absolutely.
Did you have any specific ones in mind that have caused us challenges in the past? I was planning to do a find ~/.atom/packages -name '*.node' to find some good candidates.

It looks like atom-ide-ui uses a few, but they're installed from prebuilts... other than that, atom/github certainly uses keytar...

@smashwilson I think Nuclide has native stuff? My own LaTex package has a native module dependency, though hardly a major community package.

Hydrogen uses 0mq which is a native dependency...
smashwilson @ hubtop ~/src/atom/apm (aw/massive-npm-version-bumps=)
$ apm install hydrogen
(node:82123) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
Installing hydrogen to /Users/smashwilson/.atom/packages ✓
smashwilson @ hubtop ~/src/atom/apm (aw/massive-npm-version-bumps=)
$ find /Users/smashwilson/.atom/packages/Hydrogen/ -name '*.node'
/Users/smashwilson/.atom/packages/Hydrogen//node_modules/zeromq/build/Release/zmq.node

Okay, I've identified a few packages as good test candidates for a variety of criteria. More are always welcome 👍

@smashwilson Cool. Did you check the PlatformIO ecosystem?

Nope! 😆 Do we test there? Is there someone we can ping (or a repo we can file an issue on) to check it out?

I think the main repo is platformio/platformio-atom-ide. Haven't followed the dependency chain all the way, but might be some native modules via platformio/platformio-node-helpers maybe.

Just noticed that git-plus has a package-lock.json file. Probably a good candidate.

Oh huh! I just noticed that npm added package dependencies to package.json in atom/atom:
  "dependencies": {
    "@atom/nsfw": "^1.0.18",
    "@atom/source-map-support": "^0.3.4",
    "@atom/watcher": "1.0.3",
    "about": "https://www.atom.io/api/packages/about/versions/1.10.0/tarball",
    "archive-view": "https://www.atom.io/api/packages/archive-view/versions/0.65.1/tarball",
    "async": "0.2.6",
    "atom-dark-syntax": "https://www.atom.io/api/packages/atom-dark-syntax/versions/0.29.0/tarball",
    "atom-dark-ui": "https://www.atom.io/api/packages/atom-dark-ui/versions/0.53.2/tarball",
    "atom-keymap": "8.2.10",
  }

Here's a preliminary set of performance data based on timing a bunch of apm commands from a cold cache:

Command
1.19.0
1.19.0-0

[ '--version' ]
822ms
686ms (-136ms)

[ 'uninstall',  'teletype',  'atom-lcov',  'atom-ide-ui',  'hydrogen',  'latex',  'github',  'git-plus' ]
16330ms
11872ms (-4458ms)

[ 'install', 'teletype' ]
5597ms
4880ms (-717ms)

[ 'install', 'atom-lcov' ]
3637ms
4281ms (+644ms)

[ 'install', 'atom-ide-ui' ]
25641ms
19351ms (-6290ms)

[ 'install', 'hydrogen' ]
71410ms
33564ms (-37846ms)

[ 'install', 'latex' ]
19115ms
13936ms (-5179ms)

[ 'install', 'github' ]
29325ms
15613ms (-13712ms)

[ 'install', 'git-plus' ]
16313ms
10738ms (-5575ms)

[ 'install', 'atom/github' ]
81752ms
40520ms (-41232ms)

[ 'rebuild', 'hygrogen' ]
1553ms
1111ms (-442ms)

[ 'rebuild' ]
1524ms
1160ms (-364ms)

Not bad, not bad.

No regressions detected across all three platforms other than that apm clean thing. :tada:

Railcars are rolled...  Perhaps this can be merged on Monday? :)


Bug: hdmiCecController.py fails to be included when core is used in ut-raft

bug in hdmCECController.py
import sys
MY_PATH = path.realpath(__file__)
MY_DIR = path.dirname(MY_PATH)
sys.path.append(path.join(MY_DIR,'/../../'))
from framework.core.logModule import logModule

from hdmicecModules import CECClientController, MonitoringType

needs to be
from framework.core.hdmicecModules import CECClientController, MonitoringType

it assumes that you've added hdmicecModules into the current path which you have not in this file.
or
sys.path.append(path.join(MY_DIR,'/../')) - would be required

This has been fixed now #115 has been merged


static image for visualization with plain background is not working

for example, this: https://javi.cartodb.com/api/v1/map/static/named/tpl_7a99951e_aba7_11e3_b843_0e10bcd91c2b/300/300.png
steps:

create a visualization with plain color baselayer
add a cartodb layer
request the static map

I guess it's because backend is not dealing properly with plain baselayers (take into account we have WMS layers too which can be translated to http ones)

note to self:

vis is not broken, save_named_map doesn't raise any error
other vis with plain bgs work (just return a null urlTemplate)
the error is coming from windshaft, so the issue is not a bad reading of layer types but somehow not storing the layer field inside the named map http layer.


Fix eslint issues

Thanks! I cant get to my usual setup to code properly atm :(

@HoeenCoder I can do these small stuff amidst exam.
Also get ready coz I am PRing trading soon :P

ok


Search SDK: Fixing Timeout and removing unneeded projects from the so…

…lution

Description
Fixes #2604
Now when you call searchClient.Indexex.GetClient(), it will propagate the Timeout value set on the searchClient's HttpClient.
FYI @markcowl: We're removing project references to TestFramework, HttpRecorder, and the ClientRuntime libraries. We prefer NuGet references over project references for the Search SDK solution for a couple reasons:

It speeds up build times.
It avoids build errors when the dependent projects change in unpredictable ways.
It avoids the dreaded "An item with the same key has already been added" build error (see https://github.com/dotnet/cli/issues/1860)

Finally, we're also bumping the version of the data plane SDK to 3.0.2 and fixing some typos.
There are no Swagger or generated code changes for this PR.

This checklist is used to make sure that common guidelines for a pull request are followed.

[X] I have read the contribution guidelines.
[X] The pull request does not introduce breaking changes.

General Guidelines

[X] Title of the pull request is clear and informative.
[X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[X] Pull request includes test coverage for the included changes.

SDK Generation Guidelines

[X] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
[X] The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as a link to the swagger spec, used to generate the code.
[X] The project.json and AssemblyInfo.cs files have been updated with the new version of the SDK.

FYI @chaosrealm @seansaleh

:dart:

@brjohnstmsft Travis build is broken due to coreclr package unavailability for Ubuntu. We do not have any ETA when this will be fixed yet. Please contact me via IM if you need this to be merged ASAP.

@hovsepm Thanks for letting me know. It's not super urgent. I'll check back with you in a few days.

@brjohnstmsft Travis is fixed and this PR is merged.

@hovsepm Thanks!


update supporters

@andreamari actually we need to also update the main page here: https://github.com/unitaryfund/unitaryhack/blob/main/index.md
Can you take care of this in a new PR?

#20


Error: "The track could not be added to the queue" even though it works

This is weird, even though I can properly request a song, I get a
{
    "error": true,
    "msg": "The track could not be added to the queue"
}

This was due to a docker configuration error.


[BUGFIX] no pointer in type case in component

Does somebody tests the component example ?

Yes, I tried it and tried it again and I did not not had any issue.
What's your issue ?
I tried your patch and the component crashes for me with that change.

It throws the error:
unexpected packet, got component:handshake

Yes, you are right. The component connects and works fine but does return an error despite the success.
The example client did not check the error returned. I fixed the example to handle the error and will merge your patch.
Thanks!

https://dev.sum7.eu/genofire/thrempp
It works already by sending and recieveing Threema TextMessages -> thank you for this library :+1:

@genofire Thanks for sharing this 😄 Glad to know it is useful.


[BUG] Set-PnPTenantSite Url Trailing Slash Issue

Reporting an Issue or Missing Feature
This is a bug
Expected behavior
When working with Set-PnPTenantSite I would expect there should be no difference if the site URL has a trailing slash or not.
Actual behavior
When running a command like Set-PnPTenantSite -url https://contoso.sharepoint.com/sites/test/ -Owners<EMAIL_ADDRESS>it would produce an error:
Set-PnPTenantSite: The managed path sites/test is not a managed path in this tenant.

However when I omit trailing slash in URL it would complete no problem.
Steps to reproduce behavior

Connect to SharePoint Online Admin Center with PnP
Run  Set-PnPTenantSite -url https://contoso.sharepoint.com/sites/test/ -Owners<EMAIL_ADDRESS>with a trailing slash in URL
Repeat the same command  Set-PnPTenantSite -url https://contoso.sharepoint.com/sites/test/ -Owners<EMAIL_ADDRESS>without a trailing slash this time.

What is the version of the Cmdlet module you are running?
Name    : PnP.PowerShell
Version : 1.7.0

Which operating system/environment are you running PnP PowerShell on?

[x] Windows
[ ] Linux
[ ] MacOS
[ ] Azure Cloud Shell
[ ] Azure Functions
[ ] Other : please specify

Fixed it, thanks for reporting.
Will be available in tomorrow's nightly !


Fix or remove offline mode

Offline mode (settings.OFFLINE) is currently broken due to some changes I made some time ago (didn't note it then).
Do we actually need it?
The offline mode just hides the login page, but does not invalidate active sessions, which would be required to have a clean "maintenance mode".
When I'm doing maintenance work, I stop resp. restart the service anyway, so an offline mode build into the app itself wouldn't help.
This might have been useful in the past, but at the moment I tend to completely remove it instead of fixing an unused feature.

As it actually happened this year that somebody submitted a Meldung from his still valid session after login was deactivated, I think not invalidating sessions is an issue :angel:


Playfair encrypt error encoding "BAAAH

We discovered that the encipher method of Playfair does not encrypt BAAAH correctly when using the key "KINGSABCDEFHLMOPQRTUVWXYZ".  It returns "CBCVLW" when it should be "CBVCLW".  However if I write the grid out:
KINGS
ABCDE
FHLMO
PQRTU
VWXYZ

BAAAH -> BAXAH -> BAXAHX -> CBVCLW

When using two online cipher calculators, the result was: CBCVBF. The two websites are: https://www.dcode.fr/playfair-cipher and https://planetcalc.com/7751/

When using two online cipher calculators, the result was: CBCVBF. The two websites are: https://www.dcode.fr/playfair-cipher and https://planetcalc.com/7751/

This error is coming here because it is being padded with X if you use V you get CBCVBF

I implemented it myself using X padding and it gave me: CBCVBF.

The plaintext BAAAH is padded with X. So we split as BA AA HX. Now BA are in same row so we rightshift as CB,next AA same characters so change to AX next A=(1,0) X= (4,2) so we take (1,2) and (4,0) which is CV again for HX H=(2,1) X=(4,2) so we take (2,2) and (4,1) which is LV.Correct me if I have made a mistake here and if so please tell me the correct steps

I think the mistake is when you padded the end of the string first. When padding a playfair plaintext, you follow the following steps in order:

Divide the string into pairs: in our case 'BAAAH' becomes: 'BA', 'AA', and 'H'. Don't worry for the single letter 'H' at the end for now.
Check every pair to determine if the pair contains one repeated letter: 'BA' is OK, 'AA' has repeated A, 'H' is OK.
Put 'X' between the two repeated letters in every pair: 'BAAAH' becomes: 'BA', 'AX', 'AH'
Here comes the step that you did first: Check if every pair contains exactly two letters. After filling 'X', the string is already good to go so we don't need to pad the end of the string.
Then the final plaintext: 'BA', 'AX', 'AH'.
You can learn more from STALLINGS: CRYPTOGRAPHY AND NETWORK SECURITY PRINCIPLES AND PRACTICE

Got it.I had misunderstood that the repeated one's second character is replaced by X.Thanks for the help and sorry for the trouble

You are welcome.


Should we archive this repo

@licquia - Since this is named "OLD" - is this repo no longer in use and should we archive it?

@goneall, +1 on archiving this to avoid potential confusion.
Friendly ping to @licquia

@goneall +1

Thanks @licquia - I'll archive this repo now


Connect the dots with observability

Right now there is no trivial way of connecting audit logs or debug logs (properly coraza logs) with the underlying requests or their consequent proxy logs (e.g. envoy logs). transaction ID is one identifier associated with the WAF transaction (aka the request in the server) and is local to the server request processing.
Allow the auditlogs to include extra information based on variables (e.g. REQUEST_HEADERS:X-Request-ID) is probably the easiest approach and does not need to happen in seclang necessarily but in the config of the WAF. A new auditlogpart X would be needed to include all these extra fields. Whenever you want to correlated a request with a transaction, look for the request ID in the audit logs.
Note: currently audit logs support printing the request headers but doing that for the sake of a single header is not only overkill but also a security concern as there is no redaction of potential sensitive information or PII.
Related https://github.com/corazawaf/coraza-proxy-wasm/issues/166

We should also consider redaction features for audit logs


Fix Markdown Errors

Add   {#manual-node-administration} and {#self-registration-of-nodes} to URLs

/assign @tanjunchen

$ ./scripts/lsync.sh content/zh/docs/concepts/architecture/nodes.md
diff --git a/content/en/docs/concepts/architecture/nodes.md b/content/en/docs/concepts/architecture/nodes.md
index 7bd4b355b..de08e48d8 100644
--- a/content/en/docs/concepts/architecture/nodes.md
+++ b/content/en/docs/concepts/architecture/nodes.md
@@ -11,9 +11,10 @@ weight: 10

 Kubernetes runs your workload by placing containers into Pods to run on _Nodes_.
 A node may be a virtual or physical machine, depending on the cluster. Each node
-contains the services necessary to run
-{{< glossary_tooltip text="Pods" term_id="pod" >}}, managed by the
-{{< glossary_tooltip text="control plane" term_id="control-plane" >}}.
+is managed by the
+{{< glossary_tooltip text="control plane" term_id="control-plane" >}}
+and contains the services necessary to run
+{{< glossary_tooltip text="Pods" term_id="pod" >}}

 Typically you have several nodes in a cluster; in a learning or resource-limited
 environment, you might have just one.

Thank you for proposing this fix. However, we will not merge partial fixes as per stated in the guideline.

Thank you for proposing this fix. However, we will not merge partial fixes as per stated in the guideline.

OK I'll start a new PR


Front page updates

[x] The BC logo should be clickable and bring you to provincial website: https://www2.gov.bc.ca/gov/content/home
[x] All BC Registry of Historic Places need to be replaced with BC Register of Historic Places
[x] Need short description of BCRHP for subtitle on 1st scroll page: "The BC Register of Historic Places records historic places that have been officially recognized by the provincial, federal or local / regional governments for their heritage value."

Tab in browser still says BC Registry... everything on the actual page is good.

Fixed.


No option to update user details via admin services

You can create users using the UserAdmin admin service. Lets say you want to update a claim in that user, you cannot do so as there is no method to update a user info via admin services

You can use RemoteUserStoreManagerService to update user profile.
https://localhost:9443/services/RemoteUserStoreManagerService?wsdl


Stream.pipe should give clearer error message

If undefined is passed to pipe, it sends "TypeError: Cannot call method 'on' of undefined". It would be better with a friendlier error message.

This could be expanded to checking where appropriate that the object being passed is the right type, like we do in _.pluck. For _.pipe, we could do something like in _.pipeline where there's this check:
!_.isStream(start) && !_.isFunction(start.resume)

There are also other functions that accept highland streams where a check should probably be made using just _.isStream.

Would something like this at the beginning of the function be enough?
Stream.prototype.pipe = function (dest) {
    var self = this;

    if (dest == null || !(_.isStream(dest) && _.isFunction(dest.write))) {
        throw new Error('Argument passed to pipe function must be a stream, argument provided: ' + typeof dest);
    }

Also, I noticed that if you pass undefined or a string or anything that isn't actually a number into _.take it just returns the first element of the stream. That seems a bit strange to me. I wonder should we not do a parseInt and throw or push an error if we get a NaN?


Mongo stub fix

Added MongoDate::toDateTime() metod, which returns a DateTime object representing the MongoDate object.

Your help is mostly appreciated, and improves experience for everyone, but we legally obliged to get a contributor license agreement for any change. Please follow CONTRIBUTING.md before sending any future pull requests!
this was fixed as part of our weekly stub fixing routine, so this is now obsolete


Cache-control and browser behaviour

issue left open after pull-request #906.
"chrome seems to ignore the header for some resources (but it's send), e.g. the bundle, and always ignores them on new tabs."
probably requires reading up on how the browsers handle these headers.

Fixed


How to modify XML files properly?

Hi,
I'm a student and I'm trying to use MuJoCo in an RL project.
I need to modify some of the mujoco environments.
I have tried to modify the XML files but the results are not proper.  For example, I want to modify half_cheetah so that its hind legs are smaller than the original half_cheetah.
The problem I am facing is that when I train this new model using ppo,  the hind legs are not attached properly. A screenshot of the trained agent is given below.  I would appreciate it if you could guide me on how to resolve this issue. Would I need to use solidworks to modify the XML file correctly?

You need to also modify the pos attribute of the <body> element containing the geom that you just shortened.

I did change it, and the given picture is what I get after modification. Probably I am not changing the positions correctly. I would appreciate it if you guide me on that.


Configure Parcel Aggrigation Tool is saved without relative paths

As is the script tool is looking to C:\Git...

Fixed in commit
https://github.com/Esri/community-parcels-python/commit/df963ffa3212b65e850755dae554a6dcf6eac299


Merge with upstream

This is notably handling merge conflicts with https://github.com/bytecodealliance/wasmtime/pull/6443

Thanks!


Collect envoy logs for failed CI runs

Please describe the Improvement and/or Feature Request
Currently when a CI run fails, envoy logs are not collected. As a result its very hard to understand the reason for a failure especially when its not reproducible locally.
Scope (please mark with X where applicable)

New Functionality          [ ]
Install                    [ ]
SMI Traffic Access Policy  [ ]
SMI Traffic Specs Policy   [ ]
SMI Traffic Split Policy   [ ]
Permissive Traffic Policy  [ ]
Ingress                    [ ]
Egress                     [ ]
Envoy Control Plane        [ ]
CLI Tool                   [ ]
Metrics                    [ ]
Certificate Management     [ ]
Sidecar Injection          [ ]
Logging                    [ ]
Debugging                  [ ]
Tests                      [ ]
CI System                  [X]
Project Release            [ ]

Could you expand a little bit more about what experience you're envisioning with logging? Should this be an osm CLI command or do you want a script that gets executed at the end of the CI run that dumps all the envoy proxy logs into a file which is available after the CI runs?

Could you expand a little bit more about what experience you're envisioning with logging? Should this be an osm CLI command or do you want a script that gets executed at the end of the CI run that dumps all the envoy proxy logs into a file which is available after the CI runs?

This issue is to do with the CI system where we do not collect logs from all containers on failures.


[upstream] Releasing Kogito Operator 0.12.0

Signed-off-by: Ricardo Zanini<EMAIL_ADDRESS>This is an update for the Kogito Operator on Kubernetes
Thanks submitting your Operator. Please check below list before you create your Pull Request.

Flat operator directory structure is obsolete from 23-rd of October 2019, only nested directory structure will be accepted.

New Submissions

[ ] Does your operator have nested directory structure?
[ ] Have you selected the Project Community Operator Submissions in your PR on the right-hand menu bar?
[ ] Are you familiar with our contribution guidelines?
[ ] Have you packaged and deployed your Operator for Operator Framework?

[ ] Have you tested your Operator in all supported installation modes?
[ ] Is your submission signed?

Updates to existing Operators

[x] Is your new CSV pointing to the previous version with the replaces property?
[x] Is your new CSV referenced in the appropriate channel defined in the package.yaml ?
[x] Have you tested an update to your Operator when deployed via OLM?
[x] Is your submission signed?

Your submission should not

[x] Modify more than one operator
[x] Modify an Operator you don't own
[x] Rename an operator - please remove and add with a different name instead
[x] Submit operators to both upstream-community-operators and community-operators at once
[x] Modify any files outside the above mentioned folders
[x] Contain more than one commit. Please squash your commits.

Operator Description must contain (in order)

[x] Description about the managed Application and where to find more information
[x] Features and capabilities of your Operator and how to use it
[x] Any manual steps about potential pre-requisites for using your Operator

Operator Metadata should contain

[x] Human readable name and 1-liner description about your Operator
[x] Valid category name1
[x] One of the pre-defined capability levels2
[x] Links to the maintainer, source code and documentation
[ ] Example templates for all Custom Resource Definitions intended to be used
[ ] A quadratic logo

Remember that you can preview your CSV here.
--
1 If you feel your Operator does not fit any of the pre-defined categories, file an issue against this repo and explain your need
2 For more information see here

Seems that CI hasn't finished. :(
Weird 'cause both stages have successfully finished:
https://travis-ci.com/github/operator-framework/community-operators/jobs/359995074
https://travis-ci.com/github/operator-framework/community-operators/jobs/359995073


Update Full Stack Nanodegree to v4.0

Hi Mike!
I appreciate your work :+1:
Summary of changes I made with Full Stack Web Developer Nanodegree:

Changed the order of courses to strictly follow the syllabus of v4.0
Added Google Maps APIs course
Added 2 elective courses

Intro to JavaScript
Networking for Web Developers

Removed 4 courses

Responsive Images
Intro to Backend
Developing Scalable Apps in Python
Linux Command Line Basics

I know All of that because I have taken the nanodegree and I have graduated from it :smiley:

Should the old courses be kept in the list?
I feel they should be kept in the list with some marker that tells that they are deprecated but were once part of the nanodegree.

Yes the old courses should be kept in the list :)


Discussion not working despite link in "New Issue" screen

Hi @Wulfheart , this is fixed , thx for the feedback , have a nice day !


Can't restore the original touchbar

I tried out pock and decided it wasn't for me. I uninstalled the app and now can't seem to get the original touchbar back. I downgraded to an older version and it still didn't work.

Hello, have you tried going to System Preferences -> Keyboard, and trying different settings under "Touch Bar shows" or ticking "Show Control Strip?" Hopefully, that solves your problem.


How to add authenticate users via a username and password using passport-local

Does anyone add this feature?  I add some code to server.js, but not work.
Also how to limit user join room only after login? Thanks
async function setupLocalAuth()
{
// Configure the local strategy for use by Passport.
//
// The local strategy require a verify function which receives the credentials
// (username and password) submitted by the user.  The function must verify
// that the password is correct and then invoke cb with a user object, which
// will be set at req.user in route handlers after authentication.
passport.use(new LocalStrategy(
function (username, password, cb) {
db.users.findByUsername(username, function (err, user) {
if (err) { return cb(err); }
if (!user) { return cb(null, false); }
if (user.password != password) { return cb(null, false); }
return cb(null, user);
});
}));
// Configure Passport authenticated session persistence.
//
// In order to restore authentication state across HTTP requests, Passport needs
// to serialize users into and deserialize users out of the session.  The
// typical implementation of this is as simple as supplying the user ID when
// serializing, and querying the user record by ID from the database when
// deserializing.
passport.serializeUser(function (user, cb) {
	cb(null, user.id);
});

passport.deserializeUser(function (id, cb) {
	db.users.findById(id, function (err, user) {
		if (err) { return cb(err); }
		cb(null, user);
	});
});

app.use(passport.initialize());
app.use(passport.session());

// Define routes.
app.get('/auth/login',
	(req, res) => {
		//logger.warn('login  %o',req.query.state);
		//logger.warn('login  %o',req.query);
		//const state = JSON.parse(base64.decode(req.query.state));
		const { peerId, roomId } = req.query;
		res.render('login',{peerId:peerId,roomId:roomId});
	});

app.post('/auth/login',
	passport.authenticate('local', { failureRedirect: '/auth/login' }),
	async (req, res,next) => {
		try
		{
			//const state = JSON.parse(base64.decode(req.query.state));

			//const { peerId, roomId } = state;
			const peerId = req.body.peerId;
			const roomId = req.body.roomId;

			req.session.peerId = peerId;
			req.session.roomId = roomId;

			let peer = peers.get(peerId);

			if (!peer) // User has no socket session yet, make temporary
				peer = new Peer({ id: peerId, roomId });

			if (peer.roomId !== roomId) // The peer is mischievous
				throw new Error('peer authenticated with wrong room');

			//logger.info('[login user: "%o"]', req.user);
			//logger.info('-----------------------------');
			//logger.info('[peer: "%o"]', peer);
			//logger.info('-----------------------------');

			let _userinfo = {};
			_userinfo.name = req.user.username;
			if (req.user.emails.length > 0)
				_userinfo.email = req.user.emails[0].value;
			_userinfo.picture=null;

			if (typeof config.userMapping === 'function')
			{
				await config.userMapping({
					peer,
					roomId,
					userinfo : _userinfo
				});
			}

			peer.authenticated = true;

			logger.info('[peer update: "%o"]', peer);
			logger.info('-----------------------------');

			res.send(loginHelper({
				displayName : peer.displayName,
				picture     : peer.picture
			}));
		}
		catch (error)
		{
			return next(error);
		}
	});

}

The process is as below:

after click "Log in" button, open a new browser window (or tab)

submit with correct username and password
server will call
app.post('/auth/login',
passport.authenticate('local', { failureRedirect: '/auth/login' }),
async (req, res,next) =>
4)then call (authenticate ok)
res.send(loginHelper({
displayName : peer.displayName,
picture     : peer.picture
}));
at this time, user is loged in (the Login button is green now)
click "AUDIO AND VIDEO" button to join room
But the react page is refreshed and the access address is changed to "https://<IP_ADDRESS>:3443/?displayName=Jack"

I don't know why the react page is refreshed after authenticate. Please help. Thanks.

The reason is that, the user info that I used in passport-local is different with OAuth2. This will result in websocket close which will refresh the page. The last question, how can I limit user create or join room only after login?
Boolean(socket.handshake.session.passport) &&
Boolean(socket.handshake.session.passport.user)
)
{
logger.info('socket.handshake.session.passport: "%o"',socket.handshake.session.passport);
logger.info('socket.handshake.session.passport.user: "%o"',socket.handshake.session.passport.user);
			const {
				id,
				displayName,
				picture,
				email,
				_userinfo
			} = socket.handshake.session.passport.user;

Did you have a look at the userMapping and accessFromRoles function examples in https://github.com/edumeet/edumeet/blob/master/server/config/config.example.js
This function is called for each user and you can do there whatever you want.
I think for your question the first example should do:
userMapping : async ({ peer, roomId, userinfo }) =>
{
peer.addRole(userRoles.MODERATOR);
peer.addRole(userRoles.AUTHENTICATED);
},
in combination with:
accessFromRoles : {
// The role(s) will gain access to the room
// even if it is locked (!)
[BYPASS_ROOM_LOCK] : [ userRoles.ADMIN ],
// The role(s) will gain access to the room without
// going into the lobby. If you want to restrict access to your
// server to only directly allow authenticated users, you could
// add the userRoles.AUTHENTICATED to the user in the userMapping
// function, and change to BYPASS_LOBBY : [ userRoles.AUTHENTICATED ]
[BYPASS_LOBBY]     : [ userRoles.ATHENTICATED ]
},

I find the reason of redirect after login . ( https://<IP_ADDRESS>:3443/?displayName=jack)
app.all('*', async (req, res, next) =>
{
if (req.secure || config.httpOnly)
{
let ltiURL;
		try
		{
			ltiURL = new URL(`${req.protocol }://${ req.get('host') }${req.originalUrl}`);
		}
		catch (error)
		{
			logger.error('Error parsing LTI url: %o', error);
		}

		if (
			req.isAuthenticated &&
			req.user &&
			req.user.displayName &&
			!ltiURL.searchParams.get('displayName') &&
			!isPathAlreadyTaken(req.url)
		)
		{

			ltiURL.searchParams.append('displayName', req.user.displayName);

			res.redirect(ltiURL);


DropDown window is not showing properly on 2nd level of Mat Dialog

Bug, feature request, or proposal:
Drop down is not properly showing in Mat Dialog.
What is the expected behavior?
Dropdown in 2nd level Mat Dialog should appear in 2nd Dialog Formfield.
What is the current behavior?
Dropdown in 2nd level Mat Dialog is showing outside both Dialogs.
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
I've tried Google Chrome latest Update.

Closing this issue as it does not follow the issue template. Please file a new issue that follows the template, which gives the team the information needed to investigate.


Fix keras imports

Had the same issue as #48 Fixed keras import to make it work.

@datitran  Can you take a look at the PR? Also, how do we incorporate this in the build process so that we can have a pipeline running against TF 2.0 ?

Thanks for the PR. I tested it and it works. I will merge the PR. @bugra I will upgrade to TF 2.0 soon and we will use tf.keras instead.


feat(anta.inventory): Add support for DNS names, eAPI port and device alias names

When using a containerlab topology on macOS, we need to use port forwarding to localhost to access eAPI endpoints.
The current inventory only supports IP addresses. The PR is to support hostnames (DNS names instead of IP), ports and alias names in the inventory.
The following inventory works with this PR:
anta_inventory:
  hosts:
  - name: spine1
    host: localhost
    port: 44301
  - name: spine2
    host: localhost
    port: 44302
  - name: leaf1
    host: localhost
    port: 44311
  - name: leaf1
    host: localhost
    port: 44312
  - name: leaf3
    host: localhost
    port: 44313
  - name: leaf4
    host: localhost
    port: 44314

Closing this pull request as the eAPIUrl model is not used anymore with asyncio framework


Set *.png and *.ico to be treated as binary files.

The change b732beba6 makes all files text files and sets lf as eol. This will automatically change all files to have lf if they are touched by git (e.g. via git status). This change cannot be stashed and makes it hard to work with the repo (rebase and checkout don't really work). See also #6183.
Here, we set the offending files (*.png and *.ico, but that might be more in the future) to be treated as binary files and not be changed by git.

The repo will be unusable with branches until this is merged. This is the most critical PR rn IMO.

Might also need to include *.icns in the binary, as the following file is listed as modified: macapp/assets/icon.icns.

Might also need to include *.icns in the binary, as the following file is listed as modified: macapp/assets/icon.icns.

Interesting, it wasn't flagged for me. But I added it now.

The repo will be unusable with branches until this is merged. This is the most critical PR rn IMO.

@FellowTraveler You might be able to temporarily override that behavior until this is merged by overriding the git attributes locally: https://stackoverflow.com/a/50950870/12345656

The reason .gitattributes was added was to resolve an issue with the go linter. Instead, I think we can partially revert the previous change to .gitattributes and explicitly mark *.go as LF.
e.g.
* text=auto
*.go text eol=lf

I'm happy to merge this change if you make this adjustment

Ah - glad to see someone has a solution to this, it caused me headaches when trying to update a PR yesterday, I ended up doing all sorts of hacks to work around it which eventually sent Github crazy and I had to bin the PR and create a new one!


How to set custom styles for the button?

.vue-js-switch{

     position: absolute;
     top:40%;
     z-index:1;
}

.vue-js-switch .v-switch-core{
    background-color: #2980b9!important;
}

.vue-js-switch .v-switch-button {
   border-radius: 0px!important;
}

Only the vue-js-switch class worked.  Even after using !important, there seems to be no change.

i assume you have tried with child combinator .vue-js-switch > .v-switch-core?

Same here. v-switch-input works, nested elements cannot be targeted via

@tombalev. If you are using single file component, try writing the custom styles under a separate


Added index parameter on selection handler.

Hello,
This was helpful to me, so I could associate a custom list with the search item list. Hope it helps.
Let me know if it should have any correction.
Regards,
H.

@apasccon do you think you may include this to the official pod?


Tee or die

Removes the logger which could lead to too much buffering and :boom:. Fixes #11
cc @emilymcafee it might be cool to try and retrofit one of your pending migrations to field-test this before we merge.

yup I'll test this on my next go-around :star2:


Failed to start video decoder  In Jetpack Compose

I am using RtspSurfaceView in jetpack compose like this
DisposableEffect(Unit) {
    onDispose {
        // Stop the video when the composable is disposed
        rtspSurfaceView?.stop()
    }
}
AndroidView(
    modifier = Modifier
        .fillMaxSize()
        .aspectRatio(16 / 9f),
    factory = { context ->
        RtspSurfaceView(context).apply {
            init(uri, username, password)
            start(requestVideo = true, requestAudio = false)
            setStatusListener(rtspStatusListener)
            debug = true
            rtspSurfaceView = this
        }
    }
)

I  get "Failed to start HARDWARE video decoder (The surface has been released)" & "Failed to start video software decoder. Exiting..."  errors when I  close and reopen clients multiple times.
How can I get it work in Jetpack Compose

Try to use RtspImageView instead.


regression introduced by new doc system

in case my comment on https://github.com/stevengj/PyPlot.jl/commit/ddaa060478d511de6b460f324851e93551f2a97d goes unnoticed, there is now an incompatibility with julia v0.3.  all works fine on v0.4:
julia> using PyPlot
ERROR: wrong number of arguments
 in include at /usr/bin/../lib64/julia/sys.so
 in include_from_node1 at /usr/bin/../lib64/julia/sys.so
while loading /home/arthurb/.julia/v0.3/PyPlot/src/PyPlot.jl, in expression starting on line 404

Thanks!


Integrate NIP-32 for product and stall (store) reviews

https://github.com/nostr-protocol/nips/pull/532
Not yet merged, but this NIP will allow for any kind of review, not just for things like moderation

Thank you for pointing this out. I have added a  comment to the PR.


What is the right way to have client side code run?

So, I have a component called <HomeHero/>, which is basically a big hero text with a background <video> tag.
I want to place some code that interacts with the <video> tag in the DOM (plays it, sets speed, e.t.c) once we render.
From what I understand there's some sort of gotcha about server vs. client, and not putting client side code upon first render.
Could you clarify the right place to place client side JS in components?

componentDidMount is only called on the client once the DOM node has been created. That's your place. You'll want to get the <video> DOM node via refs.

Thank you! This is working great. Loving this setup so far. Serious productivity so far.


missing promotion in gaussjacobi for integer α, β

This works:
julia> gaussjacobi(500, 0.0, -1/2);

but changing 0.0 to an Int gives an error, probably indicating a missing call to float(α) or similar:
julia> gaussjacobi(500, 0, -1/2)
ERROR: MethodError: no method matching asy1(::Int64, ::Int64, ::Float64, ::Int64)
Closest candidates are:
  asy1(::Integer, ::Float64, ::Float64, ::Integer) at ~/.julia/packages/FastGaussQuadrature/BRLTf/src/gaussjacobi.jl:200
Stacktrace:
 [1] jacobi_asy(n::Int64, α::Int64, β::Float64)
   @ FastGaussQuadrature ~/.julia/packages/FastGaussQuadrature/BRLTf/src/gaussjacobi.jl:183
 [2] gaussjacobi(n::Int64, α::Int64, β::Float64)
   @ FastGaussQuadrature ~/.julia/packages/FastGaussQuadrature/BRLTf/src/gaussjacobi.jl:45
 [3] top-level scope
   @ REPL[98]:1

jacobi_asy should be restricted to Float64 as the choice of boundary terms is precision dependent:
https://github.com/JuliaApproximation/FastGaussQuadrature.jl/blob/8b964b712a4beb4403b984311da59748af2d7b70/src/gaussjacobi.jl#L175
It would be easy to add a promote for the case of differing types of course.

Maybe promote(float(α), float(β))... since passing integers should be allowed.


Automatic PR from RestAPI-PR1671

Generated from RestAPI PR: https://github.com/Azure/azure-rest-api-specs/pull/1671

Codecov Report

Merging #1451 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1451   +/-   ##
=======================================
  Coverage   54.62%   54.62%
=======================================
  Files        3142     3142
  Lines       79379    79379
=======================================
  Hits        43362    43362
  Misses      36017    36017

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f95d529...4ac41f7. Read the comment docs.


"External Org" users getting improperly categorized as Known Guests

While I expect this issue may have broader ramifications, the scenario in which it was detected was that we have external users that are in the same org as the full CCB users.  This is being determined down at line ~240 of the slCore library -> ProcessEntitlements:
			If isReader And (externalNAB Or userNote.externalOrg(0) = "true") Then
				userNote.KnownGuestNote = "Set at line 241 and IsUserReport = " + CStr(IsUserReport)
				userNote.KnownGuest = "true" 'DNEXT-23857
			End If

However, this code was consistently returning nearly all users as known guests due to a strange malfunction much earlier in the code, ~190 of the same function:
				accessLevel = enNote.HighestAccessLevel(0)
				'Select Case enNote.HighestAccessLevel(0)
				Select Case accessLevel
				Case "6": aclLevel="Manager"
				Case "5": aclLevel="Designer"
				Case "4": aclLevel="Editor"
				Case "3": aclLevel="Author"
				Case "2": aclLevel="Reader"
				Case "1": aclLevel="Depositor"
				Case "0": aclLevel="No Access"
				Case Else: aclLevel=""
				End Select
				userNote.accessLevel = accessLevel
				userNote.acllevelcalculated= aclLevel

The accessLevel variable was my troubleshooting to see if for some reason it poorly handled the item off the document and the two saved fields at the bottom are for extracting debug-- the rest of the code functions identically.
As it runs now, the accessLevel-- highest observed access level as extracted from entitlements.nsf-- was being properly observed-- but somehow incorrectly calculated into the aclLevel that would be used for downstream CCB/CCX/Known Guest calculations.  From a calculation running this original logic above, the user is observed having Manager access to its highest access database:

However it somehow ends up with the Else, null value when this is converted via the select case:

Commenting out the Case Else, we see a dramatic swing in quantity of CCB vs. Known Guests: due to a template giving -Default- Manager access nearly everyone gets calculated as the former.  The logic is still misfiring in some way, though-- while the accessLevel is observed as 6:

The calculated access level still comes out as Designer, which we should have expected from an accessLevel of 5:

So it seems that aclLevels are somehow being calculated at levels lower than what they should be in that code around line 190.  I don't see the actual fault in the code so it's possible this is an issue with client execution of the code-- I have  observed on Notes 14.0FP1 and 14.0FP2.

Resolved in V1.2.6


Initial cleanup of gam resoldcustomer/resoldsubscription

I still have some big work to do on this function so let's hold on this for now.

I changed customer_id to customerId to make the variable match the API argument as per your request on the smime code yesterday.
I cleaned up the error messages, and handled cases where a mistyped command line argument caused an infinite loop.
The rest of the changes are bug fixes.
If you apply this PR now, I won't have to redo the work in the future and you're working from a better base.
Ross
--
<EMAIL_ADDRESS>On Feb 4, 2017, at 7:02 AM, Jay Lee<EMAIL_ADDRESS>wrote:

I still have some big work to do on this function so let's hold on this for now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

It was to late to merge this since I'd already worked on adding plan change functionality and other changes but I incorporated most of your changes in my recent commit. Regarding customer_id vs customerId, I broke from the preference because I was concerned a customerId variable would conflict with the global setting (e.g. my_customer) but that doesn't seem to be the case.

I'll watch for that commit.


Couple of small fixes

It's a start!

@renesas-rz I fixed the signal specification. I tested with dash and bash. Thanks for getting back to me!


Update location of sdkmanager in Android CI jobs

Fixes https://github.com/SFML/SFML/issues/2878
The location of the sdkmanager tool was changed in the Github Actions runners, which caused the CI to fail.
This PR updates the path to the sdkmanager program.

SFMLBuildMaster: Build this please.


Using mongodb-core with lodash

Hi all,
While trying out the MongoDB native driver for node.js we received an interesting error when trying to connect to our Mongo instance hosted on mLab:

MongoError: failed to connect to server [.mlab.com:] on first connect [MongoError: connect ECONNREFUSED <IP_ADDRESS>:27017]

What was really weird about this error is it did not occur when connecting via Mongoose using the same uri and options.  We finally tracked down the root cause today: in mongodb-core/lib/utils.js the assign() function toggles between the native Object.assign() function and the default method used for pre-ES6 versions of node.  We are also using lodash, and that package was replacing the native implementation of Object.assign with that package's version of the method.  The end result is the properties sent to the Server's connection Pool were not getting merged correctly, and it tried to connect to the Mongo instance running on "localhost:27017" instead,
The band-aid I applied was to use the pre-ES6 function exclusively and never use Object,assign.  This problem does not occur in Mongoose because it does not use lodash internally, so the behavior of Object,assign() is not affected.  Has anyone encountered a similar issue when working with lodash and mongodb-core as well in node version  4.0+?  Is there a recommended workaround in this situation that we should use instead of making a new util method that always uses the built-in assign()?
Note that I tested with the following versions of each package and the problem persisted as long as lodash is installed:

mongodb: 2.2.31 (mongodb-core 2.1.15), 3.0.4+ (mongodb-core 3.0.4+)
lodash: 3.10.1, 4.11.14

Hi @rgmembreno, if that's really what's going on that sounds like very serious bug in lodash (though this seems unlikely). In the 3.x series of the driver we have removed the custom assign implementation, in favor of always using Object.assign, and are unlikely to go back. Are you sure something else isn't polyfilling Object.assign? Can you perhaps try to use just the pieces of lodash that you need, rather than bringing the entire module?

Hi @mbroadst thanks for the quick reply!
It would be a big change to import only a handful of packages from lodash, and we don't know offhand how long it will take.  It also looks like the problem in lodash is they assume that every core function is available with prototype..call(), but assign() does not appear on the prototype as of node v 6.12.x  We're sure it's lodash because every time we hit an "Object.assign" in mongodb-core it opens up lodash while debugging.
So we should fork mongodb-core and restore the old behavior?  In the meantime we can try moving off of lodash modules that mutate object properties, but as mentioned above it requires a long time to double check our work.

Hi @mbroadst
Great news, thanks to your pointers we were able to determine that lodash was not polyfilling the Object.assign() method automatically,  It took a lot of effort but we finally found the other 3rd party module that was doing this polyfill and updated it to use the native node v6 version of the function, and now everything is working great.
Thank you for all your help with this issue--we never could have figured out the polyfill problem until you asked us to check out the lodash source code to file the bug!  We're all good here, and thanks for answering all of our questions!


Prevent lndhub:// URLs from working

If someone tries to connect an lndhub it currently allows it and then the app wallet is in a broken half working state.
Prevent lndhub:// URLs from working with handy error message directing them to the bolt hub docker github

Updated the isValidNodeAddress function in lightning-custodian-wallet. It's now calling /getinfobolt api call to the hub. If this returns with a right response we know this is a bolt hub.
You will have to rebuild the hub on your server as I added a new api call on the hub.
https://github.com/boltcard/boltcard-wallet/blob/e1cc39c38953f52983de54ed751b1cf9d1d5e90b/class/wallets/lightning-custodian-wallet.js#L618


is_pinned() returns true even after :Unpin

To reproduce:

:PinBuffer
:Unpin
:lua print(require'stickybuf'.is_pinned())

The last command returns true.

Fixed. Thanks for the report!


Rectangle tool in image editor creates rectangles slightly off from original starting point

Describe the bug
When using the rectangle tool, the first corner is placed at position slightly off (1-2 pixels) in either or both directions. Using the cursor coordinates as a guide for the starting point before clicking and then hovering over the first corner after releasing, you can see a discrepancy in the placement.
To Reproduce
Steps to reproduce the behavior:

Add the 'set background image' block to the editor and click on the image
Click on the rectangle tool in the image editor.
Navigate to your desired starting spot and note the coordinates. Make your rectangle.
Hover over your starting spot and see that the coordinates are off from your intended starting spot.

Expected behavior
The rectangle should start at the point indicated by the cursor coordinates right before clicking.
Screenshots

Desktop (please complete the following information):
Chrome

Won't fix. We have lived with issue for long enough and no customer complaints.