[Feature request] Support GPTQ quantization

So I have a GPTQ llama model I downloaded (from TheBloke), and it's already 4 bit quantized.  I have to pass in False for the load_in_4bit parameter of:
model, tokenizer = FastLlamaModel.from_pretrained(

because if I don't, I get an error thrown saying:
The model is already quantized with gptq. You can't quantize it again with bitsandbytes

But, if I pass in False for load_in_4bit, this code makes bnb_config be None:
        bnb_config = None
        if load_in_4bit:
            bnb_config = BitsAndBytesConfig(
                load_in_4bit              = True,
                bnb_4bit_use_double_quant = True,
                bnb_4bit_quant_type       = "nf4",
                bnb_4bit_compute_dtype    = dtype,
            )

and that makes quantization_config be None as well:
quantization_config = bnb_config,

and that crashes here:
        if hasattr(self, "quantization_config"):
            output["quantization_config"] = (
                self.quantization_config.to_dict()

with the error message:
'NoneType' object has no attribute 'to_dict'

So I'm not sure how to LoRA train this llama model.  Any thoughts?

I tried adding:
[...] and self.quantization_config is not None:

to the end of that line there (and similar additions in two other places that came up), and it hasn't crashed, but it's now taking a very long time to load the model, so maybe it's doing some unwanted conversion?

Yeah, it finally 'loaded' but then it said some weights of the model checkpoint were not used when initializing LlamaForCausalLM, and it listed a giant list of weights, which I'm guessing was all of them.
The the LoRA training crashed with:
Cannot copy out of meta tensor; no data!

So something definitely did not go well.

@araleza Oh no I don't think GPTQ models are supported as of yet

Currently only QLoRA via bitsandbytes is supported, hence all the error messages. If GPTQ is a super popular request, I will add it in - the dequantization steps will just be replaced, but I will have to read up on how GPTQ does it internally.
For now, is it possible to use a non GPTQ quantized model?

For now, is it possible to use a non GPTQ quantized model?

I don't know actually... I've only done LoRA training with oobabooga's Training tab, and it can only do LoRA training with unquantized models, or GPTQ models (which you have to load with the Transformers loader).  So I don't know how to load a quantized model of any format except GPTQ onto my GPU.  Any tips for which format to use instead, but still have it fit on my 24GB GPU?

@araleza Would it be possible to try load a non quantized model, then pass load_in_4bit = True via Unsloth? It should load on ur CPU / RAM then it quantizes then loads it into the GPU

I'll see for a future release if I can add GPTQ support!

I was atually just reading up upon HQQ (half quadratic quantization) https://github.com/mobiusml/hqq and maybe I'll be adding HQQ instead of GPTQ since HQQ has no need for data calibration, whilst GPTQ does.

Sounds good. I think you've got two groups of people who want to use your software:

people who have a big model and big training data, and want the fine tuning to be faster
people with 24GB cards who want to train larger models, but without quantizing them so badly that the training is meaningless.

Supporting HQQ would help the people in group 2, like me.

@araleza Cool I'll get on with HQQ! It seems like even Mixtral can supposedly fit on a 24GB card!
But HQQ supports 8, 4, 3 and 2 bit quantization so it'll be pretty useful!

@danielhanchen happy to pitch in with quantization (or other feature requests).  let me know how best to contribute!

@jeromeku More than happy to collaborate! I was actually taking a look at GPTQ the other day - I guess technically Unsloth can add in GPTQ during training - we we need is to port the dequantization kernels from GPTQ to float16 / bfloat16, and if that works, then GPTQ will be supported.
For all, I'm using bitsandbytes's dequantization kernels.
Again more than happy to collaborate if you're interested!

@danielhanchen
That should work -- this is what QLoRA does under the hood for non-LoRA weights right?  I.e., dequantizes 'frozen' weights to f16 / bf16 in order to pass grads through non-LoRA weights.
I can take a crack at this if you're more keen working on hqq...

@jeromeku I'll investigate GPTQ's dequant kernels as well! But if you're interested in adding GPTQ support - I'm more than happy for a few more OSS collaborators!
Essentially in terms of the main gist of things:

Find how GPTQ dequantizes its quantized weights to float16 / bfloat16
Extract this functionality from say Huggingface internals or some other provider like Exllama / llama.cpp etc
Replace fast_dequantize with GPTQ equivalent kernels
Fix up a few lines where Linear4bit naming conventions are seen with GPTQ equivalent conventions.
If 3 works as is, then Unsloth is now GPTQ compatible!

If you wanna take a crack at that - I'll be super grateful! In fact just step 1 or 2 is enough for a general GPTQ integration!

@danielhanchen
Will work on it!

@jeromeku Great! If you need any help - ask away! I guess we can use this Github issue as a central discussion area. I'll see if I have some time on GPTQ - probably next week ish - I'm trying to work on some other stuff currently.
Again thanks!

@danielhanchen
Trying to understand design decisions / coding style of the library.
What is the purpose of patching {Mistral, Llama}_fast_forward when initializing Mistral (pre_patch)?  It seems you are extracting sections directly from the original HF implementations of these layers (which already support flash-attn2) and in some cases using xformers for some of the ops.
Why the use of pass after every function?  This is (AFAIK) a rather unconventional python coding style?

@jeromeku prepatch essentially just patches some portions of each function to call their relevant efficient implementation - ie as you mentioned some xformers some FA2.
Oh ye sorry on my coding style - I came from like like C++ / C background so I generally like all functions / if / for loops etc to be "enclosed" to make it "look" compartmentalized.
But you can have whatever coding style you like - for eg I like spaces between eqals during variance assignments, whilst general style is var=2 and not var = 2. It definitely comes from my C background!!
If you're contributing code - I don't mind on style - that's the least of worries! :)) You can use any style you desire - it just has to work :)

@danielhanchen
Any tools / tests you use to check the correctness of gradient implementations?

@jeromeku Oh lol what I do is to get HF to do training, copy paste the training losses to Google Sheets, then with ur updated gradient implementation, log if the new training loss is mostly identical.
Another approach is to use torch.dist or torch.all_close on W.grad and new_W.grad to confirm the gradients. You'll have to  do loss.backward(Y) for eg to get the gradients.

@danielhanchen
Ok, was wondering if there was a more efficient way to do this verification.  Was trying to use torch.autograd.gradcheck but runs into issues with large inputs / outputs and mixed precision since it needs to realize the full VJP during numerical / analytical gradient calc.
I've adapted GPTQ code to re-implement fast_lora custom fwd / bwd and should have the rest done by early next week.
A minimal way to check the gradient is being calculated correctly -- akin to a unit test -- without having to do a training run would be a worthwhile effort both for existing and future implementations.

@jeromeku Actually I did technically make some functions to check gradients somewhere - I manaully made some random inputs and some random outputs, then backpropagated with torch.backward(outputs), and checked every item's .grad to confirm it - I just need to find where I wrote it :))

@danielhanchen
I wrote a small test script to do gradient checking:
import torch
from datasets import load_dataset

# 4bit pre quantized models we support for 4x faster downloading!
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from torch.utils.data import DataLoader

from unsloth import FastLanguageModel

DTYPE = torch.float16

def get_model(
    model_id="unsloth/mistral-7b-bnb-4bit",
    reference=True,
    max_seq_length=2048,
    dtype=torch.float16,
    load_in_4bit=True,
    init_lora_weights=False,
    upcast=True,
):
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name=model_id,
        max_seq_length=max_seq_length,
        dtype=dtype,
        load_in_4bit=load_in_4bit,
    )

    lora_config = LoraConfig(
        r=16,
        target_modules=[
            "q_proj",
            "k_proj",
            "v_proj",
            "o_proj",
            "gate_proj",
            "up_proj",
            "down_proj",
        ],
        lora_alpha=16,
        lora_dropout=0,
        bias="none",
        task_type="CAUSAL_LM",
        init_lora_weights=init_lora_weights,
    )

    if reference:
        model = prepare_model_for_kbit_training(
            model,
            use_gradient_checkpointing=True,
            gradient_checkpointing_kwargs={"use_reentrant": True},
        )
        model = get_peft_model(model, lora_config)
    else:
        config = lora_config.to_dict()
        del config["task_type"]
        model = FastLanguageModel.get_peft_model(
            model,
            use_gradient_checkpointing=True,
            random_state=3407,
            max_seq_length=max_seq_length,
            upcast=upcast,
            **config,
        )

    return model, tokenizer

ref_model, _ = get_model(dtype=DTYPE)
test_model, _ = get_model(dtype=DTYPE, reference=False)

def check_grad(model, dtype, seed=0, scale=1):
    wrapped_model = model.model.model
    embed_layer = wrapped_model.embed_tokens
    self_attn = wrapped_model.layers[0].self_attn
    mlp = wrapped_model.layers[0].mlp
    torch.manual_seed(seed)

    with torch.autocast(device_type="cuda", dtype=dtype):
        # embeddings = embed_layer(inputs)

        embeddings = torch.randn(
            1, 1, embed_layer.weight.shape[1], dtype=dtype, requires_grad=True
        ).cuda()
        print(f"Attention input dtype: {embeddings.dtype}")
        attn_out, *_ = self_attn(embeddings)
        print(f"Attn out dtype: {attn_out.dtype}")
        mlp_out = mlp(attn_out)

        torch.manual_seed(seed)
        fake_grad_output = scale * torch.randn(mlp_out.shape, dtype=torch.float32).to(
            mlp_out.device
        )
        mlp_out.backward(fake_grad_output)

    return mlp_out, mlp, attn_out, fake_grad_output

mlp_out_ref, mlp_ref, attn_out_ref, fake_grad_ref = check_grad(ref_model, dtype=DTYPE)
print(
    "Grad check after reference backwards:",
    test_model.model.model.layers[0].mlp.down_proj.lora_B.default.weight.grad,
)
mlp_out, mlp, attn_out, fake_grad = check_grad(test_model, dtype=DTYPE)

ref_type = torch.float32
print()
print(
    f"Checking fake grad (dY): {torch.allclose(fake_grad.to(ref_type), fake_grad_ref.to(ref_type))}"
)
# torch.max(torch.abs(fake_grad.to(ref_type) - fake_grad_ref.to(ref_type)))
# torch.allclose(mlp_out.to(ref_type), mlp_out_ref.to(ref_type))

print(f"Checking mlp grads:")
for (n1, m1), (n2, m2) in zip(mlp.named_parameters(), mlp_ref.named_parameters()):
    if "lora" in n1 and "lora" in n2:
        n1 = ".".join(n1.split(".")[:2])
        print(f"{n1}")
        print(
            f"Mean grad:\n  UNSLOTH: {m1.grad.max():.10f}\n  REF: {m2.grad.mean():.10f}\nMax abs diff: {torch.max(torch.abs(m1.grad - m2.grad)):.10f}\nMean abs diff: {torch.mean(torch.abs(m1.grad - m2.grad)):.10f}"
        )
        print()

print("Checking attn grads:")
for (n1, m1), (n2, m2) in zip(
    ref_model.model.model.layers[0].self_attn.named_parameters(),
    test_model.model.model.layers[0].self_attn.named_parameters(),
):
    if "lora" in n1 and "lora" in n2:
        # torch.allclose(m1.grad.to(dtype), m2.grad.to(dtype))
        n1 = ".".join(n1.split(".")[:2])
        print(f"{n1}")
        print(
            f"Mean grad:\n  UNSLOTH: {m1.grad.max():.10f}\n  REF: {m2.grad.max():.10f}\nMax abs diff: {torch.max(torch.abs(m1.grad - m2.grad)):.10f}\nMean abs diff: {torch.mean(torch.abs(m1.grad - m2.grad)):.10f}"
        )
        print()

Note: there are small inconsistencies between prepare_model_for_kbit_training in unsloth vs. huggingface peft when doing QLoRA fine-tuning -- peft upcasts all non-INT-8 params to fp32 -- see here.
I added an upcast kwarg to unsloth FastLanguageModel.get_peft_model that is passed to prepare_model_for_kbit_training  to replicate this behavior:
def prepare_model_for_kbit_training(
    model: Any,
    use_gradient_checkpointing: bool = True,
    use_reentrant: Optional[bool] = True,
    upcast=False,
) -> Any:
    """
    Calculates where to place the gradient checkpoints given n_layers.
    We also freeze all other layers's gradients

    Args:
        model: Any LlamaModel with layers.
        use_gradient_checkpointing (`bool`, *optional*):
            Default enabled. Provides memory savings by not saving all activations,
            but only some.
        use_reentrant (`bool`, *optional*):
            https://github.com/pytorch/pytorch/blob/main/torch/utils/checkpoint.py#L354
            Optimal gradient checkpointing algorithm which will be the default in
            future Pytorch versions.
    """

    # Freeze all parameters
    for param in model.parameters():
        param.requires_grad_(False)

    # Cast non INT8 parameters to fp32
    if upcast:
        for param in model.parameters():
            if (param.dtype == torch.float16) or (param.dtype == torch.bfloat16):
                param.data = param.data.to(torch.float32)

    if use_gradient_checkpointing:
        model.gradient_checkpointing_enable()

    # If use_reentrant = True which is the Pytorch default, we just make the input requires_grad.
    if use_reentrant:
        if hasattr(model, "enable_input_require_grads"):
            model.enable_input_require_grads()
        else:

            def make_inputs_require_grad(module, input, output):
                output.requires_grad_(True)

            model.get_input_embeddings().register_forward_hook(make_inputs_require_grad)

    return model

Here is the output from running the above script:
Checking mlp grads:
gate_proj.lora_A
Mean grad:
  UNSLOTH: 0.0441589355
  REF: 0.0000020351
Max abs diff: 0.1207160950
Mean abs diff: 0.0097856047

gate_proj.lora_B
Mean grad:
  UNSLOTH: 0.0051155090
  REF: 0.0000001698
Max abs diff: 0.0086461902
Mean abs diff: 0.0002924677

up_proj.lora_A
Mean grad:
  UNSLOTH: 0.0850219727
  REF: -0.0000299520
Max abs diff: 0.1020736694
Mean abs diff: 0.0135316616

up_proj.lora_B
Mean grad:
  UNSLOTH: 0.0048866272
  REF: -0.0000000757
Max abs diff: 0.0068296790
Mean abs diff: 0.0002973406

down_proj.lora_A
Mean grad:
  UNSLOTH: 0.0928344727
  REF: -0.0000352956
Max abs diff: 0.2047328949
Mean abs diff: 0.0073212739

down_proj.lora_B
Mean grad:
  UNSLOTH: 0.0037288666
  REF: 0.0000003116
Max abs diff: 0.0040407181
Mean abs diff: 0.0002820148

Checking attn grads:
q_proj.lora_A
Mean grad:
  UNSLOTH: -0.0000000000
  REF: -0.0000000000
Max abs diff: 0.0000000000
Mean abs diff: 0.0000000000

q_proj.lora_B
Mean grad:
  UNSLOTH: 0.0000000000
  REF: -0.0000000000
Max abs diff: 0.0000000000
Mean abs diff: 0.0000000000

k_proj.lora_A
Mean grad:
  UNSLOTH: -0.0000000000
  REF: -0.0000000000
Max abs diff: 0.0000000000
Mean abs diff: 0.0000000000

k_proj.lora_B
Mean grad:
  UNSLOTH: -0.0000000000
  REF: 0.0000000000
Max abs diff: 0.0000000000
Mean abs diff: 0.0000000000

v_proj.lora_A
Mean grad:
  UNSLOTH: 0.1055297852
  REF: 0.1329345703
Max abs diff: 0.1655731201
Mean abs diff: 0.0144135132

v_proj.lora_B
Mean grad:
  UNSLOTH: 0.0139694214
  REF: 0.0166625977
Max abs diff: 0.0193632841
Mean abs diff: 0.0024413881

o_proj.lora_A
Mean grad:
  UNSLOTH: 0.1630859375
  REF: 0.1149902344
Max abs diff: 0.1842651367
Mean abs diff: 0.0191203523

o_proj.lora_B
Mean grad:
  UNSLOTH: 0.0102157593
  REF: 0.0053596497
Max abs diff: 0.0119572878
Mean abs diff: 0.0010805393

Thoughts?

@jeromeku Great work! Some pointers:
torch.manual_seed sadly does not actually work on GPUs - torch.cuda.manual_seed is the one you want!!
torch.randn can also take device = "cuda" - so I guess my first point of manual_seed is irrelevant since ur copying from CPU to GPU
Yep one issue is the upcasting to float32 which is one of the optimizations we found for VRAM reduction.
You can see there are error differences - mainly due to Flash Attention - Pytorch does Q @ K.T and other attention ops in float16, whilst FA upcasts internally to fp32, which makes it more equivalent to full float32 training - hence the error differences.
I think the reference model you used does not have FA enabled.
But ye - great work again - super useful script :)))

@danielhanchen
What do you consider permissible range of gradient discrepancies between the unsloth and the reference HF implementation?
I.e., there are differences (e.g., up_proj) that are on the same order of magnitude as the mean grads themselves -- can this be chalked up to the use of f32 vs f16...

@jeromeku Ye one of the issues I found as well when verifying Unsloth vs normal HF - thats what I for now opted to just compare training losses directly

@danielhanchen
Just wanted to give a quick update:

I have a working implementation of gptq fast_lora working.

I patched in a triton quantized matmul kernel into the existing fused forward / backward layers
Training works and is the losses are on par with the default HF gptq fine-tuner (the non-fused, torch-only GPTQ fine-tuning model if you provide a gptq quantized model to the standard from_pretrained loader).
However, the training runs are slower than the default HF model (and also the unsloth bnb version).

Need to do some additional profiling / debugging to see where the problems are and whether a torch.compile version of the quantized matmul kernel outperforms the triton kernel.

@jeromeku Super great work! Are you testing it on a Tesla T4 or Ampere based GPU? I found older GPUs Triton kernels to be noticeably slower.
Also I found through experimentation instead of writing 1 full fused kernel for matrix mult and dequantization, to split it into 2. The dequant step should only take 1-2ms, whilst the matrix mult takes 30ms or so. The compiler can be "confused" on the dequant steps, causing it to not optimize correctly, so I found using torch.matmul to be most effective.

@jeromeku Oh ok cool! If I have to guess, it's that NVCC / the Trtion compiler is not optimizing "properly" - also did u use the matmul Triton autotuner? It could be that maybe?

@danielhanchen
Yes - used a custom autotuner that is essentially the same as default triton matmul autotuner.  Without the autotuner, performance is even worse.

@jeromeku Ohh ok ok interesting - I'm just guessing somewhere the compiler is not optimizing the dequantization parts properly

Did some preliminary profiling using torch.profiler of 4 implementations:

Default huggingface GPTQ peft tuning, which defaults to the auto_gptq cuda kernel which in turn defaults to a torch-only implementation
Huggingface GPTQ model but with auto_gptq triton quant linear layers replacing default quant linear layer
unsloth GPTQ fast_lora implementation which fuses triton quant matmul with LoRA adapters
unsloth bitsandbytes fast_lora implementation

All were 4-bit Mistral models ("TheBloke/Mistral-7B-v0.1-GPTQ" and
"unsloth/mistral-7b-bnb-4bit") running a sample batch of data for 10 iterations (5 warmup, 5 active) using float16 as torch.autocast dtype.

Going to play around with optimizing these custom autograd.Functions to see what other juice can be squeezed from triton quantized matmul kernels which are already autotuned.
Also will torch.compiling to see what kind of fused implementations it can spit out
Additional analysis using nsys also

Will draft PR the profiling script and documentation along with current fast_lora gptq implementation once cleaned up.

@jeromeku LOVEE the detailed profiling!!! Just love it!! Great work again. Interesting so the Unsloth BnB kernels run in around 3.34s whilst HF's GPTQ runs in 6.2s. HF GPTQ with your Triton patch is 8 ish seconds, and Unsloth with your Trtion patch is 6.8 seconds.
Very interesting results! Did you manage to test a GPTQ just dequantize kernel, but with Unsloth? I can see in Unsloth, matrix multiplies are taking 26% of all time, whilst GPTQ is 13% Unsloth Triton is 3% (looks like overhead?) and HF + Triton is 1.5%. The goal is to move the majority of the time over to matrix multiplies in order to leverage the GPU's Tensor Cores :))
But anyways I love the table and results and fabulous work!

@danielhanchen
Yes -- there seems to be some overhead issues with the unsloth triton quant / dequant kernels.
Just opened a draft PR with the changes.


Add nowork trigger

What do you mean it doesn't work? Is it unemployed?
What happens when you try to run it? What's the output? What's the error message? What did you want or expect to happen?

Can you rephrase it to be a bit more inviting?  We've been trying to have Liona's messages be on the friendlier side lately.

Does this sound better?

What do you mean it doesn't work? What happens when you try to run it? What's the output? What's the error message? What did you want or expect to happen? Please provide more details so that we can properly help you.

A little too soft-gloved for my tastes. But if that's what we're aiming for, so be it.

Something more concise maybe like "Saying something doesn't work or is broken is unfortunately unhelpful to those who want to help.  Please provide full error messages, ..." - much of the info in !help can be reused.

Whatever. Just close this and do it however you like.
Just pointing out that we should have a standard response to "it doesn't work".

Sure.  We typically just use !help for it but I'm all for having a specialized one.  I just want to keep it respectful since we really don't want to turn people away.


labby devices hangs when run twice

Steps to repro:

labby devices: works
labby devices again: hangs

This is actually a bug with the serial controller. We're trying to start the same thread multiple times. The serial controller thread should never end, even if a communication error with the serial port occurs.


Add YOLOv3 training

@hics33 Nice YOLOv3 training code and examples.
Can you make a PR with this commit if you ready?
Thanks!!


Spike - Adding phone number to duplicates

Precursor to 939
Spike to:

pull numbers for how many additional duplicate sets this would flag in HBA to get a sense of impact
outline a plan for testing performance on this

Switch to Backend ticket:

Update algorithm logic for phone numbers
Test on large data sets
Performance test

Time box to 3 days

https://docs.google.com/spreadsheets/d/1uJ6j_5r3IShGILOXn6to2RgEx4R2kFcDDAkRYmBdngo/edit?gid=0#gid=0

Findings:

33.5% increase in AFS Possibilities
205 new flags sets based on the phone number
521 increase in combination flag sets, 207 and 320 decrease in email and nameAndDOB flag sets respectively
199 net increase in total flag sets
Performance impact is negligible
Note: flag set numbers are based on the 70 active/closed listings the job has been run on since v2 started being used in HBA (2024-10-09 00:00 08:00)

https://github.com/bloom-housing/bloom/pull/4518


winget doesn't recognizes not the installed version of Toggl

Brief description of your issue
winget doesn't recognizes the version of Toggl.TogglDesktop and winget upgrade --all reinstall it every time.
Steps to reproduce
PS C:\Users\admin> winget upgrade
Name        ID                 Version Verfügbar Quelle
-------------------------------------------------------
Toggl Track Toggl.TogglDesktop Unknown 7.5.363   winget
1 Aktualisierungen verfügbar.

Expected behavior
winget should recognizes the version of Toggl.TogglDesktop
Actual behavior
winget doesn't recognizes the version of Toggl.TogglDesktop
Environment
PS C:\Users\dago.PRISMA> winget --info
Windows Package Manager v1.1.13405
Copyright (c) Microsoft Corporation. Alle Rechte vorbehalten.

Windows: Windows.Desktop v10.0.19044.1466
Paket: Microsoft.DesktopAppInstaller v1.16.13405.0

The infinite reinstall upgrade issue was resolved in https://github.com/microsoft/winget-cli/pull/1765, although I don't know when it will be in a release.
This is caused by the developer not putting a version number in the Add and Remove Programs table. To make this a better experience, if you can reach out to the developer and ask them to add their version number to the table it would fix the problem cleanly. There will probably eventually be mitigations for this in winget's local package tracking db, but the best fix is on the developer's side.
https://github.com/microsoft/winget-cli/issues/1692 and https://github.com/microsoft/winget-cli/issues/1439 are issues tracking this.

Ok, thanks!


DICOM format - KeyError: 'PLETH'

The DICOM format set the IOD based on the channel names. It has an entry for Pleth but not PLETH. We need to either add PLETH or move towards using case insensitive matching. Here is an example of the error:
./waveform_benchmark.py -r /home/briangow/chorus/data/mimic_iii/30/3033467/3033467 -f waveform_benchmark.formats.dicom.DICOMHighBitsChunked
________________________________________________________________
Format: waveform_benchmark.formats.dicom.DICOMHighBitsChunked
Record: /home/briangow/chorus/data/mimic_iii/30/3033467/3033467
         39540 seconds x 3 channels
         14827500 timepoints, 14786000 samples (99.7%)
________________________________________________________________
Channel summary information:
 signal       fs(Hz)     Bit resolution       Channel length(s)
 PLETH        125.00     0.00392(NU)          39507
 II           125.00     0.00787(mV)          39428
 V            125.00     0.00787(mV)          39353
________________________________________________________________
Traceback (most recent call last):
  File "/home/briangow/chorus/repos/chorus_waveform/./waveform_benchmark.py", line 6, in <module>
    waveform_benchmark.__main__.main()
  File "/home/briangow/chorus/repos/chorus_waveform/waveform_benchmark/__main__.py", line 122, in main
    run_benchmarks(input_record=opts.input_record,
  File "/home/briangow/chorus/repos/chorus_waveform/waveform_benchmark/benchmark.py", line 114, in run_benchmarks
    fmt().write_waveforms(path, waveforms)
  File "/home/briangow/chorus/repos/chorus_waveform/waveform_benchmark/formats/dicom.py", line 472, in write_waveforms
    channel_table = self.create_channel_table(waveforms)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/briangow/chorus/repos/chorus_waveform/waveform_benchmark/formats/dicom.py", line 165, in create_channel_table
    iod = CHANNEL_TO_DICOM_IOD[channel]
          ~~~~~~~~~~
```~~~~~~~~~~^^^^^^^^^
KeyError: 'PLETH'

Fixed on format-dicom branch.  Channel names internally are converted to uppercase.  The written DICOM files use the original channel name.   On reading, the requested channel names are internally converted to uppercase for channel name matching; the output uses the channel name in function call and may have different cases than what's in the DICOM file.

Fixed by https://github.com/chorus-ai/chorus_waveform/pull/77


import cycle detection test WIP / POC

Not sure if this is more trouble than it's worth at a project of our size, but...
And not even sure that this is the best way to do this, even if we do want it...
This is a test that ensures that no circular imports are present in the project.
Feedback welcome.

Apparently the best off-the-shelf dependency tool for python is snakefood, but its documentation is kind of thin: http://furius.ca/snakefood/
Probably worth using if we want to pursue enforcing any kind of policy wrt imports.

Doesn't flake8 detect these for us?

Nope.  We already do have one circular reference in the code (protocol.py and _protocol_definitions.py import each other).  Granted, this one is entirely expected.  (The inspiration for this change came from making the flakapp.py change, which of course is not going to make it in to master.)  If we do write a tool, it should be able to take a list of cycles to ignore.
So, questions:

Do we want this change (or, rather, a more polished form of it)?
If so, what other rules do we want to enforce besides no (unapproved) import cycles?  Possibly some layering rules (e.g. backend can't import frontend, etc.)?

It it's not a lot of work, this sounds like a useful addition. Anyone else have any opinions?

Alright, just checked in an update.  It's a lot of code, but basically runs two tests:

Checks for cycles in the module import graph.  We do have one cycle (see ImportGraphCycleChecker.cycleExclusions) presently, but we only fail on unexpected cycles.  If an unexpected cycle is detected, this test will fail with (one gets this error if one comments out the cycle exclusions):

PolicyException: Circular import reference: ga4gh/frontend_exceptions.py --> ga4gh/protocol.py --> ga4gh/_protocol_definitions.py --> ga4gh/protocol.py

Checks for unapproved imports that violate layering policy.  Layers are defined in ImportGraphLayerChecker.layers.  The rules are that modules in a layer can not import from layers equal to (provided the module is in a different moduleGroup) or above their own.  Unapproved imports will get a error like this (I had to sabotage the ga4gh code to emit this):

PolicyException: module 'ga4gh/serverconfig.py' in moduleGroup 'config' has dependency on module 'ga4gh/datamodel/reads.py' in moduleGroup 'datamodel'

This is a lot of code, but it does perform a useful task.
+1


FIX-#3203: Fix hanging of Modin XGBoost

Signed-off-by: Alexey Prutskov<EMAIL_ADDRESS>
What do these changes do?

[x] commit message follows format outlined here
[x] passes flake8 modin
[x] passes black --check modin
[x] signed commit with git commit -s
[x] Resolves #3203
[ ] tests added and passing
[x] module layout described at docs/developer/architecture.rst is up-to-date

@prutskov , what was the issue with hang?

@prutskov , what was the issue with hang?

@YarShev The issue was in claiming all CPU resources of Ray cluster by actors in case of a small amount CPUs (1-2) in cluster. But Modin internal remote functions requires at least 1 CPU for execution. Both cases were supported by next ways:

Claiming only part of CPUs by each actor (num_cpus = 0.01 instead of default 1.0).
Adding wait's calls in case single core cluster. This is need to provide sequential execution without competition between actors and Modin remote tasks for CPU=1 resource.


Adjusted front-page content for #155

Added "home/welcome" partial loaded by "home/index"
Added welcome links under organisational branding

looks good @vyruss


Small updates

Opening this PR for DEBUGGING
Needed to apply the key=lambda x:sorted(x.keys()) to sorted(), now trying to understand how to get hit_f to what it was previously.
Trying to understand hit_f better .... what needs to happen to my_list to get it back to what hit_f was previously?
Having a hard time understanding the double lambda filter at the end ....

hit_f -> features stored in ES as F0: v1, F1: v2 ... F511: v512
To compute cos similarity we need to make sure that dimensions in the vector coming from ES keep the same order.. Fields are prefixed with "F" in ES to be a valid mapping.


added xania's implementation to the bench

Hi @mihar-22, thank you for making this benchmark possible, it immediately pointed out fuctional and performance issues in my library that I needed to fix.
The implementation of @xania/state is inspired by rxjs.
I ran the benchmark on my machine many times and the conclusion is the same every time. Somehow is maverick only faster on the first column.

I also don't understand how preact's implementation is much faster.

Hey @xania!
There could be a few reasons it's faster:

It's not feature equivalent. We have a separate scheduler, roots, effect roots, effect nesting, scope tracking, and few other things. This is all overhead and it's why I've removed certain libraries from the bench inside this repo.

The CellX benchmark only looks at your library through a very narrow lens. It's mainly static and respects eagerly queueing operations and aggresively caching. It's a poor benchmark for signals libraries as we want to understand more of the dynamic behaviour. I recommend checking out a few more such as Reactively and the creation cost by S.js - I linked the to Solid adaptation.

I've made this mistake of thinking I've made something really fast but it turns I was handling the most important part - cleanups! This can be a huge memory leak and make the library unusable, so really make sure that everything is cleaned up correctly. From a quick look at Xania I can't see any clean up operations (maybe I missed it) but that could be why. You should notice a considerable slow down after implementing it.

He Rahim, my name is Ibrahim by the way.

Clearly I am new to these concepts. As xania is implementing the observable pattern I need to ask, are those generic reactivity features or are those specific to implementations using signals concept. preferably the tests should be able to validate the correctness of these features in a library, which gets me thinking, would you be open for PR's on that aspect?
Will check
Xania distinguishes between
a. constructing the graph modeling the flow of data, because the graph is just a local object, clean up is handled by garbage collector.
b. observing the data at the nodes that flows through the graph. you can subscribe to it which returns unsubscribable object for clean up. because nodes of the graph are hot observables (cache the data) we dont need any additional observers for the purpose of this benchmark to get the data from the graph.
Xania is optimized for both size and speed thanks to the graph algorithm used.
In Xania you can only compose to construct new graphs. Or create new graph from scratch which is usually not in the critical path. xania is not trying to optimize for this I guess because it's not solving uses cases I am familiar with. Can you think of a real world use case where one needs to remove a node B in the middle of the graph after constructing it (maybe animation?)

I've found some more perf optimizations

I also tried more smaller test cases to understand the seemingly relative high overhead of xania in those cases but then I got these results:
code can be found here

results for less then 10 are kind random on my machine at least. I also added the total column which is perfectly stable in determining the rank of libs.

That's reasonable, it still aligns with the design I had in mind because we cannot just randomly remove a node in the graph. But when we do remove a node like in your example that node is well defined as part of the graph definition. And because of that I can add that as a feature without impact on perf.

In Xania you can only compose to construct new graphs. Or create new graph from scratch which is usually not in the critical path. xania is not trying to optimize for this I guess because it's not solving uses cases I am familiar with. Can you think of a real world use case where one needs to remove a node B in the middle of the graph after constructing it (maybe animation?)

It turns out that this is really useful for UI libraries, such as when hiding (and removing) element from the DOM.
For example, Solid implements a Show component by doing something like createMemo(() => condition() ? a() : b());. Two things are important for this: when condition() goes from true to false any elements created in b() need to be destroyed (cleanups), and then the entire memo should only update if a()'s deps change, not if b()'s deps change (unsubscribe from b()).

It turns out this is achieved by monadic bind <T, U>(S<T> source, f: (x: T) => S<U>)
Example
const a = new State(1);
const b = new State("even");
const c = new State("odd");
const d = a.bind(x => x % 2 === 0 ? b : c);

In this example d depends on a and conditionally on b. if the condition changes than all connection to b are cleared and new connection is added to c.

@mihar-22
Hi Rahim, I have added all the features for better comparison, I also think it is useful to have a total column in the score table.


⚠ Samsung has degraded performance

In 8f90cee, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2360 ms

Resolved: Samsung performance has improved in 526ce9c.


I Made an App for Stable Diffusion Anime Art Generation based of Maple Diffusion

Hi everyone!
This is not technically a issue though..
I just made a Stable Diffusion for Anime app based of Maple Diffusion! It's able to run 100% offline on your Apple Devices (iPhone, iPad, Mac)
The app is called “WAIFU ART AI”, it's free no ad (an experiment app for my personal study)
It supports fun styles like watercolor, sketch, anime figure design, BJD doll etc.
Uniquely supports both English and Chinese as input text to generate amazing results.
App store page: https://apps.apple.com/us/app/waifu-art-ai-local-generator/id6444585505
This app requires a 6GB RAM device, such as the iPad M1/M2, iPhone 13 Pro, iPhone 14 or iPhone 14 Pro.
Developed based on Maple Diffusion. And using AniPlus v1 as the AI model for hosting this app.
Hope you like it!

Cool! I just tried it on macOS and it seems to work well - I've added a link to your app in the README :)

Thank you so much!!! Very appreciate your sharing of the open source Maple Diffusion. It is a truly amazing project! I've learnt a lot during the making of this app as well!


Easy way to set directory index file for http_server

It would be really nice to easily set the directory index file for http_server.
Currently we have to do this:
  staticFiles.directoryHandler = (dir, request) {
    // Redirect directory-requests to index.html files.
    var indexUri = new Uri.file(dir.path).resolve('index.html');
    staticFiles.serveFile(new File(indexUri.toFilePath()), request);
  };
In the future, I'd love to be able to do this:
  staticFiles.directoryIndexFilename = 'index.html';
And we could make it also take an array of options:
  staticFiles.directoryIndexFilename = ['index.html', 'home.html'];

This issue has been moved to dart-lang/http_server#18.


error while running GObuster

heiii
I'm having trouble in running gobuster.... i want to find directories in the hack the box ip addr and I'm connected through vpn
i don't know y i get connection refused while I/m connected to vpn
the screenshot of this problem is attached below
pls help me

can i get some answers pls its urgent

As the error message says the target timed out. You can either increase your timeout or you need to investigate the problem on your machine by using a proxy or wireshark.


Make TulipaPlots run from dataframes rather than the energy_problem

What and Why
Plotting should run from the output, not from the TEM internal structure.
Possible Drawbacks
No response
Related Issues
No response

Implement once we've gotten the Notebook going and it makes sense

This will no longer be relevant with the refactor https://github.com/TulipaEnergy/TulipaEnergyModel.jl/discussions/688 and Pluto #36  likely replacing TulipaPlots.


GLFW ignores an error from wglDeleteContext when the GL context is not the main thread

https://github.com/glfw/glfw/blob/d299d9f78857e921b66bdab42c7ea27fe2e31810/src/win32_window.c#L1509-L1510
At least on Windows, glfwDestroyWindow invokes wglDeleteContext. glfwDestroyWindow function must be called from the main thread, while WGL's context thread can be different from the main thread. Actually when the context belongs to an independent thread, wglDeleteContext returned an error ERROR_BUSY (170). This is not a bug in GLFW since GLFW doesn't care the returned value from wglDeleteContext, but I think this is a suspicious behavior. Thanks,

Oh, yes, it could be helpful to application developers for GLFW to check for and report that error. There is a note in the docs that applications must not do this, but this would be better.

Thanks.
I think it would be the best to have a way to destory the context explicitly on the context thread (and if the context is already destroyed, glfwDestoryWindow would do nothing for the context)

From

Note
The context of the specified window must not be current on any other thread when this function is called.

and

This function must only be called from the main thread.

, there is no way to delete a context not on the main thread.


Add service_name property to outputs/plugins resources

This allows for calling telegraf_outputs and telegraf_plugins multiple times to create different config files but restart the same service.

:+1:

This looks good :+1:


SOLR-16391: Migrate collection creating APIs to JAX-RS

https://issues.apache.org/jira/browse/SOLR-16391
Description
Solr is in the process of migrating its v2 APIs over to a JAX-RS framework. This gives us a more feature-rich framework for expressing APIs, makes it easier for the APIs themselves to be more REST-ful and standardized, and also makes it easier for Solr to integrate with tooling such as OpenAPI, which can be used to autogenerate client bindings for a variety of different languages.
But many non-JAX-RS APIs remain.
Solution
This commit migrates a handful of APIs that either directly or indirectly create collections over to the JAX-RS framework.  These APIs are somewhat unrelated, but they have a large number of overlapping parameters relating to collection-creation so it made sense to migrate them together.
This PR also makes a few cosmetic changes to these v2 APIs to bring them closer into line with the more REST-ful design we're targeting for v2.  The APIs modified by this commit, and their new forms, may be found below:

"Create Collection" POST /api/collections {...}
"Create Alias" POST /api/aliases {...}

In order to keep the v2 API in line with any potential v1 changes, this commit moves the actual API logic for collection-creation over to the v2 definition.
Tests
Still needs unit tests and there are a number of existing tests failing.
Checklist
Please review the following and check all that apply:

[x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[x] I have created a Jira issue and added the issue ID to my pull request title.
[x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[x] I have developed this patch against the main branch.
[ ] I have run ./gradlew check.
[ ] I have added tests for my changes.
[ ] I have added documentation for the Reference Guide

As this PR stands today, the new API and refactoring appear to work correctly for v1 requests, but actually fails for v2 requests.
The root of the problem is that multiple APIs now claim POST /api/collections as their endpoint.  (See the v2 backup, restore, create alias, set-alias-property, and delete alias APIs).
This wasn't a problem while the APIs all shared the legacy framework, as the legacy framework is designed to inspect a "command" indicator at the top level of the request body (e.g. `{"delete-alias": {...}}).  But moving "create" over to JAX-RS while the other APIs remain on legacy proved to be a problem: V2HttpCall sends incoming "create" requests to the legacy framework, which no longer handles the "create" command, so an error is thrown.
This isn't a problem long-term, but we won't be able to proceed here until the other POST /collections APIs are moved over to JAX-RS.

Towards the end of last week, I was able to add the "restore collection" API to this PR, which is the last of the "collection-creating" APIs I'd hoped to do in this PR.
In hindsight though, it's grown a little larger than I was expecting and I'm a little leery of merging as is: both for the onerous-ness of review, and because the size would make any test-noise harder to pinpoint.
To this end, I filed a PR this morning that gives the create-collection, create-alias, and restore-collection v2 APIs their own (unique) REST-ful paths.  Splitting them up to be on different paths removes the blocker I mentioned in my previous comment, and will allow me to split this PR into 3 separate PRs (one for each API).  I'll aim to merge those individually, and close this PR without merging.
The first of these single-API PRs, for create-collection, can be found here.  Hoping to have single-API PRs for Create-Alias and Restore shortly as well!


Schema validations stopped working when used enctype="multipart/form-data"

Form validations were working fine until I added enctype="multipart/form-data"
Once I added enctype the form validations always return error messages irrespective of the values we put in the fields.
I have used the following module in my app.js
var bodyParser    = require('body-parser');
const flash = require('express-flash-notification');
const cookieParser = require('cookie-parser');
const session = require('express-session');

then used them as
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(session({
    secret: 'xxx',
    name: 'xxxx',
    resave: false,
    saveUninitialized: true
}));

And I am fetching the validations result like this
var formValidationObject = validationResult(req);
Please guide what could be the possible issue.

body-parser doesn't handle multipart.
Use a package like multer for these content types, then pay attention to some common gotchas when using it.


Prevent page refresh after sorting

Lovely package! My only complaint is that the page refreshes after sorting. I noticed other sort packages behave like this as well so I am not sure it is even possible to fix.

Fix the question. (928113f)[https://github.com/ofcold/nova-sortable/commit/928113f78c2674bcedf3af718923054a4f9199a6]


Add IS_LOCAL to /metadata/endpoints response (fixes #372)

Fixes #372

Approved and @emerkle826 advises that the failing tests in NonDestructiveOpsIT are flakes. Merging.


Use sha instead of ref in refresh workflow

Jira ticket URL
https://dfedigital.atlassian.net/browse/TEVA-2093
Changes in this PR:
Use sha instead of ref in refresh workflow -<EMAIL_ADDRESS>

Review app deployed to https://teaching-vacancies-review-pr-3214.london.cloudapps.digital


Xarray attributes (CRS, transform) missing

I have issues opening some catalog on https://earth-search.aws.element84.com/v0.
When I try to make some operation with rioxarray (e.g., a clip operation) I get the error:
MissingCRS: CRS not found. Please set the CRS with 'rio.write_crs()'
That's because into the attributes of the xarray returned from the to_dask() method attributes transform, crs and res have not be set. I noticed this is happening only for catalogues of the beginning of 2018.
For this reason, rioxarry can't georeference data in the proper way.
Why those attributes have not been set?
A snippet of code to reproduce the error:
from shapely import wkt
import intake
import intake_stac
import satsearch

wkt_geometry = "Polygon ((9.59935220198788208 44.99328353692637705, 9.82295202809843637 44.99189113160298348, 9.82469113104509972 45.11301999240544092, 9.60061886478441018 45.11441825799879268, 9.59935220198788208 44.99328353692637705))"

start_date = "2018-01-01"
end_date = "2018-01-03"

results = satsearch.Search.search (
            url="https://earth-search.aws.element84.com/v0",
            intersects=mapping(wkt.loads(wkt_geometry)),
            datetime="%s/%s" % (start_date, end_date),
            query={"eo:cloud_cover": {"lte": 100}},
            collections=["sentinel-s2-l2a"],
        )

items = results.items()

for item in items:
        print("Processing item {} for asset {}".format(item, asset_key))
        single_item = intake.open_stac_item(item) # stac_item_collection[str(item_by_date)]

        asset_xarray = single_item['SCL'](chunks=dict(band=1, y=2048, x=2048)).to_dask()
        asset_xarray.rio.set_nodata(0)
        asset_clipped = asset_xarray.rio.clip([mapping(_wkt_bounds)], crs=4326, all_touched=True, drop=True)
        xarray_all_patches.append(asset_clipped)

    mosaic = rioxarray.merge.merge_arrays(xarray_all_patches, precision=50, nodata=nodataval)

The clip operation with rioxarray is the one that fails (asset_clipped = asset_xarray.rio.clip([mapping(_wkt_bounds)], crs=4326, all_touched=True, drop=True) )

@g2giovanni Currently intake-stac returns a standard xarray DataArray. Note that rioxarray is an extension of xarray, and the CRS is only added to the xarray object if you open a file via rioxarray or if you explicitly use the xarray_dataarray.rio.write_crs() method as the Error message indicates.
So before running methods like rio.clip() you can do something like asset_xarray.rio.write_crs(item.properties['proj:epsg'], inplace=True)
There is some discussion of using rioxarray behind the scenes in a future version of intake-stac (https://github.com/intake/intake-stac/pull/75)

@g2giovanni I'm also noticing you are using the 'sentinel-s2-l2a' collection rather than 'sentinel-s2-l2a-cogs' which is probably what you want since they are COGs rather than jpeg2k and not in a requester pays bucket
Also, have a look at pystac-client which is replacing sat-search

@g2giovanni I'm also noticing you are using the 'sentinel-s2-l2a' collection rather than 'sentinel-s2-l2a-cogs' which is probably what you want since they are COGs rather than jpeg2k and not in a requester pays bucket

@matthewhanson seems that not all L2a data is available as COGs. I actually tried this when looking into the issue here:
import intake
from pystac_client import Client # https://github.com/stac-utils/pystac-client

aoi = {'type': 'Polygon',
 'coordinates': [[[9.59, 44.99],
   [9.82, 44.99],
   [9.82, 45.11],
   [9.60, 45.11],
   [9.59, 44.99]]]}

catalog = Client.open("https://earth-search.aws.element84.com/v0")
results = catalog.search(#collections=['sentinel-s2-l2a'], #2 found
                         collections=['sentinel-s2-l2a-cogs'], #0 found
                         intersects=aoi,
                         datetime='2018-01-01/2018-01-03',
                         )

print(f"{results.matched()} items found")

Definitely some things missing here and there.

@scottyhq you're right. It's the reason why I use the "sentinel-s2-l2a" for older years.
Anyway, I think I found the problem: some geotiff file is not georeferenced properly in the collection. Actually, I bypassed the problem, georeferencing images with this kind of problem. Maybe could those files be fixed?

I think I found the problem: some geotiff file is not georeferenced properly in the collection. Actually, I bypassed the problem, georeferencing images with this kind of problem. Maybe could those files be fixed?

@g2giovanni if there is a metadata issue with the sentinel-s2-l2a-cogs you can open an issue here describing in detail https://github.com/cirrus-geo/cirrus-earth-search


Fixes a small typo in topology aware hints page

CLA signed

/check-cla

Thanks @kenaniah! This LGTM once you're able to get the CLA check to pass.

I signed it

Hey there! @kenaniah, looks like you haven't signed the CLA yet. Could I please have you do that? https://github.com/kubernetes/community/blob/master/CLA.md

/check-cla

/lgtm


Source data mini-dashboard

Fun fact: we have 5485 individual datasets (in the open data consortium way of defining them) archived in edm-recipes!
Really a bit of a POC, but just about ready to get something initial merged, and get logging a bit Check out the product of this on the qa app! Deployed it from my branch

https://de-qaqc.nycplanningdigital.com/?page=Source+Data
Don't hit scrape data, it's all scraped as of yesterday (1/30). That will just delete/re-scrape data for all datasets used as inputs to the current data product

Commit 6 still needs some testing, but hope things are fairly good to go. 6 also needs the most feedback
A lot of this revolves around a new table in edm data - edm-qaqc.source_data.metadata_logging. This contains logs for every data library run/dataset in edm-recipes. It has columns

name
version
timestamp
runner (local git username or ci job link)
event_source (more on this later)

Commits

Interestingly in the earlier dev container, it was the db that had port 8501 exposed. But apparently it worked!
As part of this, I wanted to interact with "execution_details", i.e. metadata about how code has been run outside of just library (since I want to grab that information for use in the qc app), so this logic is moved outside of library in dcpy and expanded a bit. utils.logging seemed like a good place to do this, but happy to put elsewhere
Expand on above by moving the definitions of structures in config.ymls in edm-recipes to connectors.edm.recipes
Expand logic around recipe dataset metadata, add functions around scraping this information from s3 and persisting it in EDM-DATA. Happy to walk through this. If you're curious about the "event_source", that's expanded a bit in commit 6, I'll explain there
plotting utilities, page for source data qa. Currently, just have this as a separate page, rather than including it in the pages for each data product
log when uploading via data library, and differentiate from "scraped" logs with the "event_source" column. Main rationale is this - for many "standard" updates, this is sort of irrelevent. When we scrape from scratch, we could just delete whole table, recreate from s3. However, for socrata datasources, we might load on 1/2, and see that the datasets version is from 1/1. Then when we re-run on 1/9, we look up and again see that it was last updated on 1/1, and we re-upload to s3. This is good for logging - we'll log that this was updated on 1/9. But s3 no longer has evidence of the 1/2 loading (at least for now - we could implement versioning. Maybe it'd be worth it for stuff like this). This seems maybe a little clunky/overkill for us, maybe should just keep this simple and treat all logs the same and just don't delete logs (do some sort of validation based on not adding duplicate rows based on dataset, version, timestamp when scraping s3 for metadata)
Slightly hackily copy in recipes to app docker build. Doing this keeps it decently simple as it mimics where they are in local development for consistency. Ideally eventually recipes are maybe not in product folders but that's another convo

weird cases in edm-recipes that I noticed while scraping data

dcp_cpdb:  "2023_executive_points", "2017_adopted_points", "2017_adopted_polygons" versions are all missing config.yml/json files
doitt_buildingfootprints v 20230414 - malformed execution_details
nypl_libraries - version "testor" exists, no config.yml
"test_nypl_libraries" is a dataset, no config.yml

Not sure if there's anything to do about the cpdb ones. doitt building footprints could edit manually to add timestamp field.
Test datasets ideally are moved out entirely, they shouldn't be in our production data

Looking at the metadata_logging table in postgres, there are values NULL, Manual, could not parse. Do you know in what case(s) we get one of these values?

Looking at the metadata_logging table in postgres, there are values NULL, Manual, could not parse. Do you know in what case(s) we get one of these values?

NULL are before any metadata like this was included in the config.json/yml files in edm-recipes.
Manual means run locally, if there's no name after it that means git command line call did not return a username.
Could not parse is when something went wrong generating this info - not sure if maybe this happened when the docker image was still getting used

Per discussion with @sf-dcp , I should write some tests for connectors.edm.recipes as part of this PR

Issues might come up from multiple loading of multiple file formats - need to check how I'm logging in commit 6

Issues might come up from multiple loading of multiple file formats - need to check how I'm logging in commit 6

maybe a filename column? so (name, version, filename, timestamp, runner, event_source)

Issues might come up from multiple loading of multiple file formats - need to check how I'm logging in commit 6

maybe a filename column? so (name, version, filename, timestamp, runner, event_source)

I don't think we want to see those as separate events though, I'd prefer that one gha run of data library is one row in this table

Issues might come up from multiple loading of multiple file formats - need to check how I'm logging in commit 6

maybe a filename column? so (name, version, filename, timestamp, runner, event_source)

I don't think we want to see those as separate events though, I'd prefer that one gha run of data library is one row in this table

what about a binary col for each data format? or a column containing a list of data formats

I like a list over binary columns. so a column of type array?

This has been open/tinkered with long enough that a fresh full review would be good. Commit-by-commit will certainly make the most sense


refactor: enable optional chaining

Coverage remained the same at 100.0% when pulling 829bb4cf6eaecfff25759d5c747502ad28b2ed34 on enable-optional-chaining into 5c807272d1bbd722c63c21d5f1e5c40af6e5297a on main.

:tada: This PR is included in version 1.3.2 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


Feature/fv3 obs space

Description
Adds a new needed obs space script for plotting 6-tile FV3 model files. Includes changes to existing files as well as new added files.

I'm not sure this is the correct name for the class. Rather than ModelObsSpace perhaps it should be FV3Restart?

For whatever name you choose note that we will end up adding another class that reads cubed sphere history, both GMAO and NCEP history. So we'll want to be able to distinguish between Restarts, as you have here, and History.
Are you planning to add a new transform that interpolates the cube to a generic latlon grid that can be plotted with cartopy etc?

@danholdaway I think we went with CubedSphereRestart for this class now. In terms of transforms, this plots on a map with Cartopy already, so I don't think there is a need to convert to LatLon. @kevindougherty-noaa should have an example somewhere here that he can upload that shows what the plots will look like with EVA.

Thanks for the info @CoryMartin-NOAA. I didn't realize that Cartopy could plot a cubed sphere gridded field. Good to know.

Nice! Looks good.


How do I resend a registration confirmation email

So if I register a new user and for some reason fail to confirm the signup.  When I try to login later it fails with NOTALLOWED = TRUE, which means the email/account is not verified. Keep in mind I am using the email as the username for login.  Is there a way to resend the confirmation code email?  I cant seem to find anything.  When I try to run
await _userManager.SendEmailConfirmationTokenAsync(<CognitoUser>)
It fails with an OBJECT NOT SET TO INSTANCE OF REFERENCE and I am retrieving and passing a valid user.  I do do a down and dirty way of just changing the EMAIL attribute to the same email but Im not sure that forces a new email confirmation code to be sent.  This is important as I have the flag set to not allow the user to login if the email is not confirmed, so I need to be able to let the user choose to resend one if they didnt get it or they didnt see it.

Anymore thoughts on this one?

Hi @jlwhitfill I think I narrowed down the issue, it has to do with the fact that the cognito api expects the user to be authenticated to verify the attributes. I am trying to reproduce the issue with a cognito pool similar to yours.

I have a PR in that adds an API for the purpose of resending the confirmation code for account signup.
In depth:
For Cognito, attribute verification is a different api than account signup verification. ie you can change your email and call the email verification api as you did, but this only verifies the email, not the account.
Now when calling the new api, you will get a confirmation code sent to your email/device, then need to call usermanager.ConfirmSignupAsync() with the token. This confirms the account AND the email/device at the same time.
Also note that you can call AdminConfirmSignupAsync() . This will confirm the account BUT NOT the email/device. This is when SendEmailConfirmationTokenAsync can later be used by a user with a confirmed account but not a confirmed email.
I will let you know when the new package is available on NuGet

https://www.nuget.org/packages/Amazon.AspNetCore.Identity.Cognito/<IP_ADDRESS>  should fix the issue.
The usermanager now exposes ResendSignupConfirmationCodeAsync

This works correctly on my end, thank you!

Do any plans exist to update the sample app, or at least document that SendEmailConfirmationTokenAsync  should not be used?
This is missing, and the use of the other SendEmailConfirmationTokenAsync  call that will always fail is somewhat misleading, IMHO.
I registered a user last night and got to account validation of it this morning to find the code I was sent last night no longer worked, as cognito considered the code I had been sent invalid, and any attempt to use SendEmailConfirmationTokenAsync method to send me a new code failed with an error from Cognito that was not helpful.
Hours of googling later I found this, and was able to use the ResendSignupConfirmationCodeAsync method mentioned above to send myself  a new validation token/code that I was able to use to validate the account using a validation endpoint on our side,  but that was a lot of pain that could have been fixed easily by simply not directing people to use SendEmailConfirmationTokenAsync  in the sample app if its not what we should be using.


[BUG] The wrong value of the invite bonus is displayed to all new users on PRODUCTION env.

The wrong value of the invite bonus is displayed to all new users on PRODUCTION env.
Steps to reproduce:

create a new user
observe the result

Actual result:
The wrong value of the invite bonus is displayed to all new users on PRODUCTION env. (2125G$ instead of 100G$)
Expected result:
The value of the invite bonus is displayed properly on PROD (100G$ instead of 2125G$)
Environment:
PROD 1.35.20
Devices:
Windows 10 // Google Chrome v96
Samsung S20 FE // Android 11
Note:
To me its happens to every new user that I created
Attachments:

@sirpy this information is coming from the database.

@TomasGonzalez
the issue is here
inviteFriendsMessage.data.readMore = inviteFriendsMessage.data.readMore
          .replace('100', bounty)
          .replace('50', bounty / 2)

we replace 100 with 250
then we replace 50 with 250, this also replaces the 50 of the "250"
we need to use other place holders in the invite card. like INVITERBOUNTY instead of 100 etc...


add footer

One change I can make before this pr is accepted is migrating the css in the .css file to tailwind, let me know if you would like me to make those changes

Changed title, PR didn't really line up clearly with the UI revisions. Here is a screencap of the latest footer. The github icon redirects to this repo

@Dgarc359
i dont think this should be footer it should be a section for subscibing to news letter .
can you setpreate the button from the input
add a border or try to play with baground to make the input field clear kinda the contrast kinda wierd

I agree. to be honest it should be just after the call to action in the hero section for right now maybe? I'll leave it up to you too. it's really coming together!

has conflicts now.

I'll resolve those and make the button changes in the next commit. Thanks for the heads up

Latest picture:

Not sure why linting is failing, I don't see many details on that

Here's latest look


Updated some language files

Closes #848

Looks like this branch needs rebasing too.

Squashed, rebased and good to go!


Heading should be at the centre of the screen

In mobile view: Heading should be at the centre of the screen

@samagragupta I done it,
by change the text-align: center in main.css

@chetan-v I could not see your Pull request.


KAFKA-7819: Improve RoundTripWorker

This patch changes the Trogdor RoundTripWorker to use a long field for maxMessages and makes the consumer group used unique

JDK 11 passed, JDK 8 failed with one test (lost track of which one it is)
retest this please

Thanks, @stanislavkozlovski .  Looks good overall.  Suggest using a mutex and condition variable rather than busy-waiting.

retest this please

LGTM

JDK 8 passed
JDK 11 failed with two tests apparently (https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/3385/console), yet the logs only show one - kafka.api.ConsumerBounceTest > testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup FAILED, which is unrelated
I think this is good to merge

Failed test is unrelated.


Make Forego port match APPLICATION_HOST port

Suspenders previously included a .foreman file which specified that Foreman should use port 3000, and not the default of 5000 (rationale in a42ec7e). The switch from Foreman to Forego in 6b7614a removed this file since Forego doesn't support it, causing a reversion to port 5000 and a mismatch with the APPLICATION_HOST and ASSET_HOST variables in the sample .env file.
Forego does support specifying a port via the PORT environment variable, so this change adds that variable to the sample .env file.
(closes #665)

@mike-burns, I remember you spotted conflicts when using the HOST env variable. This change introducing PORT env variable (after discussion in https://github.com/thoughtbot/suspenders/issues/665) looks good to me, and would like your review before merging. Thanks!

LGTM.

Thank you!


MAINT: clarify the status of the scipy sourceforge page

Orion Poplawski on ML, https://mail.scipy.org/pipermail/scipy-dev/2016-February/021233.html

I'll note that
http://www.scipy.org/scipylib/download.html#official-source-and-binary-releases
still points to the sourceforge site which does not appear to be updated.

This page also has links pointing to SF: https://scipy.org/install.html
Note also that the content on the website is not automatically updated just by merging, see the README.txt


Add sync workflow

This PR adds a GitHub Action that downloads and updates settings and metrics YAML files on demand.

@rmfitzpatrick Could you have a look?


[DOC, MRG] Add "info" to docdict

This adds consistency by referencing info from the docdict.

Thanks @alexrockhill !


Improve a few JVP rules with rewrites from Herbie.

How do efforts like this interact with XLA's optimization of orders of operations & related rewrites?

It's hard to say without looking harder at XLA. However, I think these do no harm for the moment (even if XLA undoes them!) and we can look into any precision-reducing optimizations in XLA separately.


Persist widget view cell information to nb metadata

@SylvainCorlay this is what we talked about.
cc @minrk
Right now I'm persisting this to doc level metadata, so GUIDs are in one place, but really it it's cell level information.  Should I persist it per cell instead?

Awesome thanks!

merged :)

Given the return return thing that was in the code, I think that it was one of those false positives in the tests where Promise is not defined.


chainlocks: Another attempt to fix deadlocks

POTENTIAL DEADLOCK DETECTED
Previous lock order was:
 cs_main  llmq/quorums_chainlocks.cpp:139
 (1) cs  llmq/quorums_chainlocks.cpp:139
 cs_vNodes  net.cpp:3662
 (2) cs_inventory  ./net.h:1085
Current lock order is:
 pnode->cs_sendProcessing  net.cpp:2854
 cs_main  net_processing.cpp:3946 (TRY)
 mempool.cs  net_processing.cpp:4183
 (2) pto->cs_inventory  net_processing.cpp:4183
 pto->cs_filter  net_processing.cpp:4233
 (1) cs  llmq/quorums_chainlocks.cpp:87

This deadlock was introduced in https://github.com/dashpay/dash/pull/3962, fixed in https://github.com/dashpay/dash/pull/3972 and re-indroduced in https://github.com/dashpay/dash/pull/3990/ 🙈

How do you check these? Can we lint for this?

How do you check these?

By compiling with DEBUG_LOCKORDER https://github.com/dashpay/dash/blob/master/doc/developer-notes.md#development-tips-and-tricks and running on testnet


Please update SAP clustering overview suggestions for shared storage

As of September 2016, Azure doesn't offer shared storage that you can use to create a shared storage cluster.

It is almost 2023, there are several shared storage options now, shared managed disks, Azure NetApp Files, Windows Cluster Pooling of storage drives, this should really be revisited with modern recommendations.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 4d8da06a-43ad-9b28-a88e-a9da515db7d3
Version Independent ID: 2054515f-d7cd-63e2-e6e8-5cbf2e39e0dc
Content: High availability for SAP NetWeaver on Azure VMs - Azure Virtual Machines
Content Source: articles/virtual-machines/workloads/sap/high-availability-guide.md
Service: virtual-machines-sap
GitHub Login: @rdeltcheva
Microsoft Alias: juergent

@spoelstraethan
Thanks for your feedback! We will investigate and update as appropriate.

Thank you for the feedback - the article was refreshed. The reference architectures for SAP (application layer) on Windows can be accessed by navigating to "SAP Application Layer - Windows" section in the table of contents.

@spoelstraethan
We are going to close this thread as resolved but if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.


Logo for djangocms-blog

Do we have a logo for djangocms-blog?
I am in the process of creating my own blog page and I would like to display the django-cms and djangocms-blog logos.

djangocms-blog does not have any logo, I think the django CMS logo would be enough :)
But it would be nice to start a reference to existing djangocms-blog websites: please post yours here

Will do as soon as I have something online.


Context menu for safes in the sidebar

Overview
Currently the sidebar only allows to add safes that have been loaded into the sidebar. This functionality could be expanded by allowing the user to:
(1) add safe
(2) rename safe
(3) remove safe
(4) copy address
(?)
Requirements
Screens
draft tbd

Links
Slack discussion: https://gnosisinc.slack.com/archives/CQB729LLF/p1637059391381700

As a reference, here's how Zerion has implemented this.

Done in web-core.


Feature packages graph traversal

@danielshaw42,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Preparing to publish Japanese docs

Summary
Added Japanese to the language selector
Details and comments

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.03%) to 83.213%

Totals

Change from base Build<PHONE_NUMBER>:
0.03%

Covered Lines:
8843

Relevant Lines:
10627

💛  - Coveralls


test: clean up unnecessary uses of match_exact

It was found during the review of https://github.com/rust-lang/cargo/pull/13842#discussion_r1591789376
We should be happy using assert_match_exact directly.
The extra arguments to match_exact are not necessary for those test cases.

@bors r+

:pushpin: Commit 88ac4aaf2eb174e5217d0c1de6082914bdc6f950 has been approved by Muscraft
It is now in the queue for this repository.

:hourglass: Testing commit 88ac4aaf2eb174e5217d0c1de6082914bdc6f950 with merge c77e4eaf5ae15a7581ee7761accac0cc23b0470c...

:broken_heart: Test failed - checks-actions

@bors retry

:hourglass: Testing commit 88ac4aaf2eb174e5217d0c1de6082914bdc6f950 with merge e6603b7243f5a582497db2436e65e9f18dc782e3...

:sunny: Test successful - checks-actions
Approved by: Muscraft
Pushing e6603b7243f5a582497db2436e65e9f18dc782e3 to master...


The training process detail

Hi, did u first train the projector, and then train projector + LLM, what's the detail of them.

Oh!, before using external cv models, we slightly train the projector only with 1 percent of total batch, and then we freeze it. I think it does not depend on the performance quietly, which is also evaluated in the paper MM1, apple paper.

so first trainng proj only , then free proj, and train vit and llm? Why freeze proj at second time?

This is because I did not capture its effectiveness of training proj at second stage, at least on our model. The effectiveness of training MoAI-Mixer modules seem more better.

Most of the time open ivison encoder training got worse result.
Do u think you can get better results if train them all?

I cannot convince that most of time open vision encoder got worse result. I think it depends on the model setup or training setup. Recently, I read more papers training vision encoder leads to performance gain.

The MM1 paper claims that open vit only works (better) then un free when imaga tokens are too many.
It might depends on the LLM and proj design

Yes, recent papers have had lots of image tokens by using dynamic looking image. As I said, it depends on design of models. It makes sense many image feature can represent more rich onformation for VL tasks.

Sorry to ask a maybe naive question. What does the "projector" you mentioned mean?   Is this MLP the projector? "Two linear layers with GELU activation function serve as the bridge connector between vision and language components, denoted by 'MLP'"

The projector means a few layers with MLP, of which role is to make a bride connector from vision encoder to backbone multimodal LLM

The projector means a few layers with MLP, of which role is to make a bride connector from vision encoder to backbone multimodal LLM

Thanks for your reply. I noticed that you used qlora in training. Did you use qlora in training the projector? or you just use qlora in training the following steps?

No reason to quatize projector. Thus, used backbone llm only

Thanks very much. Another question, did you use InternLM or InternLM2 as base model?

We used https://huggingface.co/internlm/internlm2-7b receiving the highest like

We used https://huggingface.co/internlm/internlm2-7b getting the highest likes

thanks very much.

Sorry to bother again, I noticed that you wrote all training is based on Llava-instruct-655k filtered by ShareGPT-4v. I want to ask that, when you train the vision projector, your mentioned 1/10 of the dataset, are they randomly selected from Llava-instruct-655k? Why didn't you use the LLAVA pretraining dataset of LCS-558K?

Yes, used randomly selected data through pytorch dataloader. It does not affect performance well. LLaVA trained the model with only training projector in the pretraining stage with pretraining data. However, technically I saw the observation that there is no need to pretrain in terms of performances, cause this may be reason that the difference between pretrain and instruct dataset is answer's length. All has instruction samples. Instead, I experienced that the most important factor to improving performances is not the number of data samples only but injected external knowledge

Yes, used randomly selected data through pytorch dataloader. It does not affect performance well. LLaVA trained the model with only training projector in the pretraining stage with pretraining data.
MM1 apple paper has also shown that no matter we choose the type of projector, there is no performance differerence.
By combining these results, we can conclude that the pretraining datset is not necessary cause llava only trained the projector with pretraining dataset.
Technically, I also saw the observation that there is no need to pretrain in terms of performances, cause this may be reason that the difference between pretrain and instruct dataset is answer's length. All has instruction samples.
Instead, I experienced that the most important factor to improving performances is not the number of data samples only (assuming we have enough number of data samples. It easily makes misunderstanding that small number of data can be perofrmant.) but injected external knowledge

Sorry to bother again. I have other questions about the training. How many cards did you use during training and how long is the training time for the two stages?

Yes, used randomly selected data through pytorch dataloader. It does not affect performance well. LLaVA trained the model with only training projector in the pretraining stage with pretraining data.
MM1 apple paper has also shown that no matter we choose the type of projector, there is no performance differerence.
By combining these results, we can conclude that the pretraining datset is not necessary cause llava only trained the projector with pretraining dataset.
Technically, I also saw the observation that there is no need to pretrain in terms of performances, cause this may be reason that the difference between pretrain and instruct dataset is answer's length. All has instruction samples.
Instead, I experienced that the most important factor to improving performances is not the number of data samples only (assuming we have enough number of data samples. It easily makes misunderstanding that small number of data can be perofrmant.) but injected external knowledge

Sorry to bother again. I have other questions about the training. How many cards did you use during training and how long is the training time for the two stages?

We use approximately 665K number of training samples, and two or three days are consumed for each training step with 6 x A6000.

Dear author, thanks a lot for your help! I have another question.
In the inference code, you make prompt with:
"prompt = " [UNUSED_TOKEN_146]user\n" + prompt + "[UNUSED_TOKEN_145]\n[UNUSED_TOKEN_146]assistant\n""
Seems a little different with llava. In training, did the prompt processing in the same way with that in inference?

Dear author, thanks a lot for your help! I have another question. In the inference code, you make prompt with: "prompt = " [UNUSED_TOKEN_146]user\n" + prompt + "[UNUSED_TOKEN_145]\n[UNUSED_TOKEN_146]assistant\n"" Seems a little different with llava. In training, did the prompt processing in the same way with that in inference?

The input prompt is important to conduct instruction tuning and it is really dependent with language model. However, the prompt i inference might not be too sensitive to generate desired answers.

mportant to conduct instruction tuning and it is really dependent with language model. However, the prompt in inference might not be too sensitive to generate desired answers.

Thanks a lot for your kind reply! That's very useful.
So, in training, you changed the system prompt to ""AI assistant should give helpful and detailed answers to user after fully understanding an image.",and keep the left setting of conversation in the same way with Llava?

mportant to conduct instruction tuning and it is really dependent with language model. However, the prompt in inference might not be too sensitive to generate desired answers.

Thanks a lot for your kind reply! That's very useful. So, in training, you changed the system prompt to ""AI assistant should give helpful and detailed answers to user after fully understanding an image.",and keep the left setting of conversation in the same way with Llava?

As i experienced, system prompt did not affect the performances well. It is just a format

nversation did not affect the performances well. It is just a format

It is quite reasonable. There is another question about the training. In inference, the cv models seems not so fast, it requires several seconds to run all the cv models for one image. How did you reach a high speed in training?


contains-string fails in eo repo

The test below is fails in eo repository.
[] > catch-contains-string
  assert-that > @
    "abpba"
    $.contains-string
      "aboba"

with error
org.eolang.ExUnset: Error at "memory.enclosure" attribute
        at org.eolang.AtNamed.get(AtNamed.java:99)
        at org.eolang.PhDefault.attr(PhDefault.java:263)
        at org.eolang.PhLocated.attr(PhLocated.java:117)
        at org.eolang.PhOnce.attr(PhOnce.java:91)

@Graur please take a look

@levBagryansky Indeed, it's a bug. Thanks!

@levBagryansky the puzzle #215 is still not solved.


Dependency array-record Not Available on ARM64 Architecture

Issue
I am currently using Orbax for checkpointing in my project, which is being developed on an ARM64 architecture (Apple Silicon). Orbax has a transitive dependency on array-record through etils. However, array-record does not provide a version compatible with ARM64, leading to installation issues when using Poetry.
Environment
Orbax version: [0.1.9]
Python version: [3.9.13]
Operating System: macOS on ARM64 (Apple Silicon)
Dependency management: Poetry
Steps to Reproduce

Include orbax as a dependency in a Python project managed with Poetry.
Run poetry install on an ARM64 Mac.

  • Installing array-record (0.5.0): Failed

  RuntimeError

  Unable to find installation candidates for array-record (0.5.0)

  at ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/chooser.py:73 in choose_for
       69│
       70│             links.append(link)
       71│
       72│         if not links:
    →  73│             raise RuntimeError(f"Unable to find installation candidates for {package}")
       74│
       75│         # Get the best link
       76│         chosen = max(links, key=lambda link: self._sort_key(package, link))
       77│

Cannot install array-record.

Observe that the installation fails due to array-record not being available for ARM64.

Connection between Orbax and array-record
> poetry show array-record --tree
array-record 0.5.0 A file format that achieves a new frontier of IO efficiency
├── absl-py *
└── etils *
    ├── absl-py *
    ├── fsspec *
    ├── importlib-resources *
    │   └── zipp >=3.1.0
    ├── numpy *
    ├── tqdm *
    │   └── colorama *
    ├── typing-extensions *
    └── zipp * (circular dependency aborted here)

> poetry show orbax --tree
orbax 0.1.9 Orbax
└── orbax-checkpoint >=0.1.8
    ├── absl-py *
    ├── etils *
    │   ├── absl-py * (circular dependency aborted here)
    │   ├── fsspec *
    │   ├── importlib-resources *
    │   │   └── zipp >=3.1.0
    │   ├── numpy *
    │   ├── tqdm *
    │   │   └── colorama *
    │   ├── typing-extensions *
    │   └── zipp * (circular dependency aborted here)
    ├── jax >=0.4.9
    │   ├── importlib-metadata >=4.6
    │   │   └── zipp >=0.5 (circular dependency aborted here)
    │   ├── ml-dtypes >=0.1.0
    │   │   ├── numpy >=1.21.2 (circular dependency aborted here)
    │   │   └── numpy >1.20 (circular dependency aborted here)
    │   ├── numpy >=1.21 (circular dependency aborted here)
    │   ├── opt-einsum *
    │   │   └── numpy >=1.7 (circular dependency aborted here)
    │   └── scipy >=1.7
    │       └── numpy >=1.21.6,<1.28.0 (circular dependency aborted here)
    ├── jaxlib *
    │   ├── ml-dtypes >=0.1.0 (circular dependency aborted here)
    │   ├── numpy >=1.21 (circular dependency aborted here)
    │   └── scipy >=1.7 (circular dependency aborted here)
    ├── msgpack *
    ├── nest-asyncio *
    ├── numpy * (circular dependency aborted here)
    ├── protobuf *
    ├── pyyaml *
    ├── tensorstore >=0.1.35
    │   └── numpy >=1.16.0 (circular dependency aborted here)
    └── typing-extensions * (circular dependency aborted here)

Extra
The issue lies within etils using array-record and array-record doesnt have a distribution for the ARM64 architecture so it fails,
https://pypi.org/project/array-record/#files
Poetry
Here is my .toml for reproducability:
[tool.poetry.dependencies]
python = ">=3.9.0,<=3.9.18"
jaxtyping = "^0.2.11"

[tool.poetry.group.mltools]
optional = true

[tool.poetry.group.mltools.dependencies]
numpy = "^1.23.1"
scipy = "^1.9.0"
einops = "^0.5.0"
hydra-core = "^1.2.0"
omegaconf = "^2.2.3"
wandb = "^0.13.5"

[tool.poetry.group.dataset]
optional = true

[tool.poetry.group.dataset.dependencies]
tensorflow-macos = {version = "^2.12.0", platform = "darwin"}
tensorflow-datasets = "^4.7.0"

[tool.poetry.group.torch]
optional = true

[tool.poetry.group.torch.dependencies]
torch = "^1.13.1"
torchvision = "^0.14.1"
functorch = "^1.13.1"

[tool.poetry.group.jax]
optional = true

[tool.poetry.group.jax.dependencies]
jax = { version = "^0.4.11", source = "jax-macos", markers = "platform_machine == 'arm64'" }
jaxlib = { url = "https://storage.googleapis.com/jax-releases/mac/jaxlib-0.4.11-cp39-cp39-macosx_11_0_arm64.whl", markers = "platform_machine == 'arm64'" }
jax-metal = { version = "^0.0.4", markers = "platform_machine == 'arm64'" }
flax = "^0.5.2"
optax = "^0.1.3"
orbax = "^0.1.9"

[tool.poetry.group.jupyter]
optional = true

[tool.poetry.group.jupyter.dependencies]
notebook = "^6.4.12"
jupyter = "^1.0.0"
ipykernel = "^6.15.1"
ipython = "^8.4.0"
requests = "^2.31.0"

[tool.poetry.group.additional]
optional = true

[tool.poetry.group.additional.dependencies]
black = {extras = ["jupyter"], version = "^22.6.0"}
pre-commit = "^2.20.0"
pytest = "^7.1.3"

[[tool.poetry.source]]
name = "jax-macos"
url = "https://storage.googleapis.com/jax-releases/jax_releases.html"
priority = "primary"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I understand that this may be more appropriate over at etils codebase but I was wondering how essential etils is. Would it be enough to use the built in pathlib library and use path when a directory is needed?


Update map.yaml

I'm sorry for late reply.
Thanks for your contribution. I receive your PR.
I change branch to develop. If you have an another PR, please request PR into develop branch.
Thanks!
Gilbert.


Unable to install

Hi All,
I've been trying to install/setup grunticon for the past 1.5hrs without much luck. Attached is the npm-debug.log file. It looks to be an issue with imagemin?
I have tried:

installing in existing web directory by itself (removed existing node_modules folder)
installing in a brand new empty directory by itself
installing other dependencies and then running npm install grunt-grunticon --save-dev

None of the above work. In most cases it will freeze and not do anything for a good 15-20min. Imagemin installs normally though.
Node version is v0.10.24 on Mac OS X (10.10.5).
I am about to try and update Node (never done this before) but I thought I'd post already in case it was obvious to anyone.
Thanks,
Jess
npm-debug.log.zip

Updating to:
node - v4.2.4
npm - v2.14.12
Fixed these issues however now I get the invalid name: '@filamentgroup/worker-farm' issue. I did read the other issues but didn't understand how to resolve the issue. Is anyone able to explain it to someone with barely any experience in command line/node?
Sidenote: if I run sudo npm install -g npm@latest it kills my npm and I get sudo: npm: command not found. If I run the installer package again it fixes that issue... am I upgrading it wrong?

@jesspritchard have you tried running npm install -g npm@latest without sudo? That can lead to some problems from time to time.

@jesspritchard that being said, this should work for users of npm 2.14 just fine.
Can you do me the favor of going into a new directory and running these three commands?
node --version
npm --version
npm install @filamentgroup/worker-farm

@jefflembeck
npm install -g npm@latest withouth sudo I get the following:
MacBook-Pro:www jesspritchard$ npm install -g npm@latest
npm ERR! Darwin 14.5.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "npm@latest"
npm ERR! node v4.2.4
npm ERR! npm  v2.14.12
npm ERR! path /usr/local/lib/node_modules/npm
npm ERR! code EACCES
npm ERR! errno -13
npm ERR! syscall rmdir
npm ERR! Error: EACCES: permission denied, rmdir '/usr/local/lib/node_modules/npm'
npm ERR!     at Error (native)
npm ERR!  { [Error: EACCES: permission denied, rmdir '/usr/local/lib/node_modules/npm']
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'rmdir',
npm ERR!   path: '/usr/local/lib/node_modules/npm' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.
npm ERR! error rolling back Error: EACCES: permission denied, rmdir '/usr/local/lib/node_modules/npm'
npm ERR! error rolling back     at Error (native)
npm ERR! error rolling back  { [Error: EACCES: permission denied, rmdir '/usr/local/lib/node_modules/npm']
npm ERR! error rolling back   errno: -13,
npm ERR! error rolling back   code: 'EACCES',
npm ERR! error rolling back   syscall: 'rmdir',
npm ERR! error rolling back   path: '/usr/local/lib/node_modules/npm' }
Running node --version results in  v4.2.4
Now running everytime I try to run npm install -g npm@latest it seems to break my npm so when I run npm --version I get  -bash: /usr/local/bin/npm: No such file or directory however after running the node update package again I get  2.14.12
Running npm install @filamentgroup/worker-farm I get
npm ERR! Darwin 14.5.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "@filamentgroup/worker-farm"
npm ERR! node v4.2.4
npm ERR! npm  v2.14.12
npm ERR! Invalid name: "@filamentgroup/worker-farm"
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     https://github.com/npm/npm/issues
Thanks for taking a look - I've read through a lot of the other issues and everything suggested I either don't understand or didn't work.

It looks like your issue at hand might be with your node/npm installation :/
That's not great news.
Now, can you tell me what happens when you run:
sudo npm --version?

@jefflembeck
sudo npm --version outputs 2.14.12
I was able to install grunticon on my Windows PC at the office. It has node 4.2.3 and npm 2.14.7
When updating node on my Macbook I used the package installer from https://nodejs.org/en/. What would you suggest to try and fix my installation? I'm very much a beginner when it comes to all this and don't have the best knowledge of OSX either :(

@jesspritchard Yeah, ok, we have a permissions issue then. This is fixable!
Try going through this: https://docs.npmjs.com/getting-started/fixing-npm-permissions

And thanks for putting up with this for us. I really hope we can help you out.

@jefflembeck
I've gone through the Option 1 steps. Is there a way to test if this fixed things?
I've tried running installing grunt-grunticon and @filamentgroup/worker-farm and still get the issues?
Thanks

Now that I've had a chance to sit down and try again (rather than trying to do it between projects) I have got it to work.
In case other people have the same problems this is what I did:
As @jefflembeck said, it did appear it was a permissions issue. I followed the instructions in the link @jefflembeck provided (https://docs.npmjs.com/getting-started/fixing-npm-permissions). After this I was able to successfully run npm install -g npm@latest. npm is now version 3.5.3
Running npm install grunt-grunticon --save-dev then successfully worked.
@jefflembeck Thanks for your help! Much appreciated.

@jesspritchard Oh that's great news! I'm glad you got it. Let us know if you run into any other problems.


incremental typing of objects

Search Terms
dictionary
Suggestion
Sometimes I want to do something like:
const a = {}
a.foo = () => {}
a.bar = () => { a.foo(); }

-> but this is hard to type because the a.bar cannot change the type of a.
You can give a type with foo? bar? but its all a mess.
I tried quora, irc and gitter -> but didn't get a reply whether this is possible.
Its a eventually a complicated topic because if a was used somewhere else and
you do 'destructive' updates such as changing types other code might be affected.
So this would require introducing tracking what is being done with it.
In cases like above it looks trivial.
Note
This could be a duplicate. Thus if you know a proposal/solution kindly point to it. I failed finding it.

This would be analogous to the way arrays are gradually typed under noImplictAny:
let a = [];
a.push(1);
a[0].toExponential // a is number[]

Duplicate #12416


addTileOverlay Zoom Level

I'm submitting a ...  (check one with "x")

[ ] question
[x] any problem or bug report
[ ] feature request

If you choose 'problem or bug report', please select OS:  (check one with "x")

[ ] Android
[x] iOS

cordova information:
com.googlemaps.ios 2.6.0 "Google Maps SDK for iOS"
com.telerik.plugins.nativepagetransitions 0.6.5 "Native Page Transitions"
cordova-plugin-camera 3.0.0 "Camera"
cordova-plugin-compat 1.2.0 "Compat"
cordova-plugin-device 1.1.7 "Device"
cordova-plugin-file 5.0.0 "File"
cordova-plugin-file-transfer 1.7.1 "File Transfer"
cordova-plugin-geolocation 4.0.1 "Geolocation"
cordova-plugin-googlemaps 2.2.9 "cordova-plugin-googlemaps"
cordova-plugin-inappbrowser 1.7.2 "InAppBrowser"
cordova-plugin-network-information 1.3.4 "Network Information"
cordova-plugin-splashscreen 4.1.0 "Splashscreen"
cordova-plugin-statusbar 2.4.1 "StatusBar"
cordova-plugin-whitelist 1.3.3 "Whitelist"
cordova-sqlite-storage 2.3.0 "Cordova sqlite storage plugin"
ionic-plugin-keyboard 2.2.1 "Keyboard"

"ionic-native/google-maps": "^4.7.0"
"ionic-native/core": "^4.7.0",

Current behavior:
There seems to be a discrepancy between the map zoom level and the zoom level that is passed to addTileOverlay(). The addTileOverlay() zoom level that is used is consistently higher than what is return by getCameraZoom(). For example, if the camera zoom is level 19 the argument that is passed to addTileOverlay will be level 20.
The tiles that are getting pulled from the server are therefore incorrect for the zoom level.
Expected behavior:
The x, y, zoom level arguments passed to addTileOverlay should reflect the actual map zoom level.
Device
iPad

The x, y, and zoom values are just passes from the native api to js code.
I guess you use 256px for tile size.
Use 512px.
This is not a bug.
https://github.com/mapsplugin/cordova-plugin-googlemaps-doc/blob/master/v2.0.0/class/TileOverlay/addTileOverlay/README.md


fix(federation): shared mutation field should not be batched

In case of shared root field on Mutation, it was batched incorrectly across subgraphs. But instead only one mutation should be called as mutations should not be parallel

@Urigo It is actually an implementation detail(causing extra operation) not a compatibility issue.


error

when I run cmake --build . --config Release, an error has shownlibcifpp/include/cif++/Cif++.hpp:35:20: fatal error: optional: No such file or directory, what's wrong?

You must have a very old C++ compiler if the standard optional header file is not available. Please upgrade your compiler, in case you use GCC I'd recommend to use at least version 9. clang version 10 or higher is also an option.

when I use gcc 9.5, an error has shown In file included from libcifpp/src/AtomType.cpp:30:0: libcifpp/include/cif++/Cif++.hpp:35:20: fatal error: optional: No such file or directory compilation terminated.

and clang must be needed?


Fixed constructor

Previous implementation led to construction of uninitialized object (read here why). As a result, all message handlers for the protocol were NULL and no incoming messages were processed.

Finding this PR could have saved me two days of debugging. I found the same Error.


[SPARK-30797][SQL] Set tradition user/group/other permission to ACL entries when setting up ACLs in truncate table

What changes were proposed in this pull request?

This is a follow-up to the PR #26956. In #26956, the patch proposed to preserve path permission when truncating table. When setting up original ACLs, we need to set user/group/other permission as ACL entries too, otherwise if the path doesn't have default user/group/other ACL entries, ACL API will complain an error Invalid ACL: the user, group and other entries are required..
Why are the changes needed?

Without this fix, TRUNCATE TABLE will get an error when setting up ACLs if there is no default default user/group/other ACL entries.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Update unit test.
Manual test on dev Spark cluster.
Set ACLs for a table path without default user/group/other ACL entries:
hdfs dfs -setfacl --set 'user:liangchi:rwx,user::rwx,group::r--,other::r--' /user/hive/warehouse/test.db/test_truncate_table

hdfs dfs -getfacl /user/hive/warehouse/test.db/test_truncate_table
# file: /user/hive/warehouse/test.db/test_truncate_table
# owner: liangchi
# group: supergroup
user::rwx
user:liangchi:rwx
group::r--
mask::rwx
other::r--

Then run sql("truncate table test.test_truncate_table"), it works by normally truncating the table and preserve ACLs.

cc @cloud-fan @dongjoon-hyun This targets both mater and branch-3.0.

Test build #118288 has started for PR 27548 at commit b2fa454.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23046/
Test PASSed.

Test build #118288 has finished for PR 27548 at commit b2fa454.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118288/
Test PASSed.

@cloud-fan That's good suggestion. Let's see if we can simplify it in next versions.

Test build #118321 has started for PR 27548 at commit 7568db7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23079/
Test PASSed.

Oh, @viirya . During deleting my comments, it seems that your reply also is gone. Sorry for that.

@dongjoon-hyun It's ok. Thanks for review!

I'll merge this since the last commit is only comment-only change due to me.
Merged to master/3.0.

Since this is a security stuff and SPARK-30312 is in branch-2.4, I'll merge this to branch-2.4 together after testing locally.

@dongjoon-hyun Ok. Thanks!

Test build #118321 has finished for PR 27548 at commit 7568db7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118321/
Test PASSed.