Not detecting any spikes on GPU cluster

Hello,
I've been testing out the SpikeInterface pipeline for 2.0 probes -- I followed the steps you all went through during the 10/7/21 meeting to get everything installed, and added in the line to filter and downsample recording_lf from recording_ap. However, when it's sent to MATLAB to run kilosort2, it doesn't detect any spikes from the preprocessed data. I've run this session through kilosort2 outside of SpikeInterface, and I know this isn't the case. A thought we had was whatever pre-processing is being done through SpikeInterface is incorrectly reading the SpikeGLX metadata, since the metadata structure for 2.0 probes is slightly different from 1.0 probes.
I've included a zip folder containing the cluster output/error stack (slurm-7949799.out), the SLURM script to submit the job (spikeinterface_job.slurm), the python script from brody-lab-to-nwb that I used to run SpikeInterface (spikeglx_spikeinterface_pipeline.py), and the metadata from one of my 2.0 probes in case the differing file structure is contributing to the issue (M136_2021_12_08_rig204_g0_t0.imec0.ap.meta)
Thanks!
Sarah Jo
SpikeInterface_GPUclusterError.zip

@alejoe91 Any ideas? Should we just suggest an update to the new API and see if the issue is reproduced there?

@alejoe91 Is likely quite busy right at the moment as they are preparing for a new SpikeInterface release tomorrow.
First question, when you say "2.0 probe" is this the same as "Neuropixels 2.0" format? If so I think there is a different extractor altogether for that, probably only available in the new interface: @alejoe91 do you know have a link to that?
In the meantime I took a look at what I could based on those error message and the issue seems like it might come up well before we reach the sorting stage (though hard to be 100% sure without the ap.bin).
It appears it could be a problem with the way the converison to microvolts is done; if we look at the metafile you've attached in the in the SpikeInterface_GPUclusterError.zip file, and compare the fields related to ap channel gains, we find they are all zero in the case of your meta file, whereas other SpikeGLX files we've worked with have a non-zero value there.
To reproduce in isolation
from pathlib import Path

from spikeextractors.extractors.spikeglxrecordingextractor.spikeglxrecordingextractor import readMeta

M136_meta= readMeta(Path("E:/Brody/Issues/M136_2021_12_08_rig204_g0_t0.imec0.ap.meta"))
M136_ap_gains = [x.split(sep=" ")[3] for x in standard_meta['imroTbl'].split(sep=")") if len(x.split(sep=" ")) >= 3]

standard_meta= readMeta(Path("E:/Brody/Chronic Rat Neuropixels (Poisson Clicks Task)/A242_2019_05_30/Raw/2019-05-30_g0/2019-05-30_g0_imec0/2019-05-30_g0_t0.imec0.ap.meta"))
standard_ap_gains = [x.split(sep=" ")[3] for x in standard_meta['imroTbl'].split(sep=")") if len(x.split(sep=" ")) >= 3]

gives

Actual extraction is done in a less hacky way than this, but these are the elements used in line: https://github.com/SpikeInterface/spikeextractors/blob/master/spikeextractors/extractors/spikeglxrecordingextractor/readSGLX.py#L222
Now, I would have thought that spike sorting might have been done on raw traces anyhow, but perhaps the gains were invoked at some point during pre-processing and that corrupted the traces after that point, so that they were all 0 or np.nan prior to entering the sorter? @alejoe91 Would know more about that

Hi guys! Yes I think that the new release will solve it. We will provide very soon a new notebook that uses the new SI API, which is much more efficient than the old one. Also NP2 should be already working out of the box with the new API, but it would be great if you could upload a full recording on the shared drive so we can make sure that it does!
Will keep you updated!

@agbondy @sjvenditto In the meantime, an intermediate fix just to keep you going for now, could be to hard-code the gains to the true value (if you know it) after initializing the RecordingExtractor. That is, in your attached script,
recording_ap = se.SpikeGLXRecordingExtractor(ap_bin_path)
# recording_lf = se.SpikeGLXRecordingExtractor(lf_bin_path)

recording_lf = st.preprocessing.bandpass_filter(recording_ap, freq_min=1, freq_max = 1000)
recording_lf = st.preprocessing.resample(recording_lf, 2500)

if stub_test:
    print("Stub test! Clipping recordings!")
    recording_ap = se.SubRecordingExtractor(recording_ap,
                                            end_frame=int(nsec_stub * recording_ap.get_sampling_frequency()))
    recording_lf = se.SubRecordingExtractor(recording_lf,
                                            end_frame=int(nsec_stub * recording_lf.get_sampling_frequency()))

recordding_ap.set_channel_gains(gains=proper_scalar_gain_value)
# recording_lf.set_channel_gains(gains=proper_scalar_gain_value)

That would at least resolve that one particular issue; no guarantees it wouldn't run into any others, but might be worth a try.
You can and should check that the traces of the object look OK after that point, since that is likely what caused the issue in the first place prior to it reaching the spike sorting level.

@alejoe91 I uploaded a recent rat Neuropixels 2.0 recording to the shared Google Drive: https://drive.google.com/drive/u/2/folders/1DWxAIMAlDJVxMKQyLMiyWCnX0v7Qi9uy
There are two folders there. One is a mouse recording that Manuel uploading some time ago. The more recent one is the one I just added, which is a useful additional example because two probe were recorded simultaneously. I don't know how the API handles that currently.

@CodyCBakerPhD Depending on the timeframe of the new SI release, we may try what you suggest. Thanks.

@CodyCBakerPhD Depending on the timeframe of the new SI release, we may try what you suggest. Thanks.

We will release today, but the pipeline will require some modifications on our side to be compatible with the new release @agbondy

Awesome. Let us know! Great to see such enthusiastic, rapid development of these tools!

Sorry for the slow follow up -- I added in the suggested line of code after the stub test and I'm still running into the same error. It seems like the error is coming from readSGLX in the spikeextrators package, which is taking the gain from the meta file regardless of what I set in the spikeinterface pipeline script

@sjvenditto Ah, true that.
(a) To continue harnessing the existing pipeline (which uses spikeextractors function calls), we can potentially make use of some existing utils functions from the new SpikeInterface.
Could you try using the following lines to load the data via the new SpikeInterface version and see if that works? (will have to pip install spikeinterface)
import spikeinterface
from spikeinterface.extractors import SpikeGLXRecordingExtractor

coerced_recording = spikeinterface.old_api_utils.NewToOldRecording(recording=SpikeGLXRecordingExtractor(folder_path="path to folder with NP2 data"))

(b) If that still doesn't enable use of the pipeline, @alejoe91 can help provide instructions for how to update all of the function calls to the new version, it's not too much trouble TBH.

Setting KILOSORT2_PATH environment variable for subprocess calls to: /home/venditto/github/brody-lab-to-nwb/scripts/../../sorters/kilosort2
Traceback (most recent call last):
  File "spikeglx_spikeinterface_pipeline.py", line 61, in <module>
    recording_ap = spikeinterface.old_api_utils.NewToOldRecording(recording=SpikeGLXRecordingExtractor(folder_path=ap_bin_path))
  File "/home/venditto/.conda/envs/spikeinterface/lib/python3.8/site-packages/spikeinterface/extractors/neoextractors/spikeglx.py", line 37, in __init__
    NeoBaseRecordingExtractor.__init__(self, stream_id=stream_id, **neo_kwargs)
  File "/home/venditto/.conda/envs/spikeinterface/lib/python3.8/site-packages/spikeinterface/extractors/neoextractors/neobaseextractor.py", line 38, in __init__
    stream_id = stream_ids[0]
IndexError: index 0 is out of bounds for axis 0 with size 0

@alejoe91 Any ideas?

Hi @sjvenditto
Can you try to load the object directly from the new API? In case that works out of the box, give us a week or so to push a new pipeline with the new API (which is also a lot faster!).
You can try the following:
From temrinal:
>>> pip install --upgrade spikeinterface

In the notebook:
import spikeinterface.extractors as se

recording = se.read_spikeglx(folder_path="path-to-spikeglx-root-folder", stream_id="imec0.ap")
print(recording)

Let me know if that works! I'll put together an updated notebook in the meanwhile :)

It looks like it works loading it in:
SpikeGLXRecordingExtractor: 384 channels - 1 segments - 30.0kHz - 7376.323s

Am I able to use this in place of the existing code for reading in recording_ap? Or is the new API incompatible with other things later in the pipeline?

@sjvenditto Please download and try out the new and improved pipeline and let us know if you run into any more trouble!
https://github.com/catalystneuro/brody-lab-to-nwb/blob/si_new_np2/notebooks/spikeglx_spikeinterface_pipeline_NEW.ipynb

I've copied the lines into a separate script to run on the cluster, and it looks like everything is loading correctly, but I'm getting hung up on the peak detector:
SpikeGLXRecordingExtractor: 384 channels - 1 segments - 30.0kHz - 7376.323s
Sampling frequency: 30000.0
Traceback (most recent call last):
  File "/home/venditto/github/brody-lab-to-nwb/scripts/spikeglx_spikeinterface_pipeline_new.py", line 57, in <module>
    peaks = scp.detect_peaks(recording_processed, **job_kwargs)
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sortingcomponents/peak_detection.py", line 227, in detect_peaks
    processor = ChunkRecordingExecutor(recording, func, init_func, init_args,
TypeError: __init__() got an unexpected keyword argument 'chunk_duration'

job_kwargs is defined the same as in the notebook:
job_kwargs = {"n_jobs": 4, "chunk_duration": "1s", "progress_bar": True}

Oups my bad! chunk_duration is a new feature that will be available in the next release. You can use instead: chunk_size=30000, or you can specify the amount of memory you want to use (e.g. toral_memory='2G'). These are used to split the recording in small chunks for parallel processing. Let me know if that's works

@sjvenditto Hey there - did the new notebook run successfully on the new data?

Apologies - I've gotten the SI side of things to run, but I ran into a kilosort installation error. I haven't been able to get around to fixing the installation, but I was hoping to get to it in the next week!

@alejoe91 Do we happen to have the kilosort docker working yet?

@CodyCBakerPhD not yet, but we're on it! Should take  some time though

I've got the ks2 installation path worked out, and this is the error I'm running into now:
Traceback (most recent call last):
  File "/home/venditto/github/brody-lab-to-nwb/scripts/spikeglx_spikeinterface_pipeline_new.py", line 94, in <module>
    sorting_outputs = ss.run_sorters(
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/launcher.py", line 257, in run_sorters
    _run_one(task_args)
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/launcher.py", line 33, in _run_one
    run_sorter(sorter_name, recording, output_folder=output_folder,
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 67, in run_sorter
    return run_sorter_local(sorter_name, recording, output_folder=output_folder,
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 86, in run_sorter_local
    SorterClass.set_params_to_folder(
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 142, in set_params_to_folder
    raise AttributeError('Bad parameters: ' + str(bad_params))
AttributeError: Bad parameters: ['chunk_mb']

I took the parameters from the notebook but un-commented the ks2 parameters:
# user-specific parameters
sorter_params = dict(
    kilosort2=dict(car=False, n_jobs_bin=12, chunk_mb=4000),
    #ironclust=dict(filter=True),
    #tridesclous=dict(n_jobs_bin=job_kwargs["n_jobs"]),
    #spykingcircus=dict(filter=True, num_workers=16),
    #herdingspikes=dict(filter=True)
)

Looks similar to above issue where this is now "chunk_duration": "1s" @alejoe91 is that correct? Or is the sorter-specific param still the same?

@sjvenditto @CodyCBakerPhD Yes! sorry we forgot to update the parameters in the notebooks o our side. In the new wrapper, you can pass the total_memory param (e.g., total_memory="2G" will use a total of 2GB RAM - see https://github.com/SpikeInterface/spikeinterface/blob/master/spikeinterface/sorters/kilosort/kilosort.py#L49). You can also check any sorter specifi param with: ss.get_sorter_params(sorter_name)

Replaced chunk_mb with total_memory. New error stack:
Traceback (most recent call last):
  File "/home/venditto/github/brody-lab-to-nwb/scripts/spikeglx_spikeinterface_pipeline_new.py", line 94, in <module>
    sorting_outputs = ss.run_sorters(
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/launcher.py", line 257, in run_sorters
    _run_one(task_args)
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/launcher.py", line 33, in _run_one
    run_sorter(sorter_name, recording, output_folder=output_folder,
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 67, in run_sorter
    return run_sorter_local(sorter_name, recording, output_folder=output_folder,
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 88, in run_sorter_local
    SorterClass.setup_recording(recording, output_folder, verbose=verbose)
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 171, in setup_recording
    cls._setup_recording(recording, output_folder, sorter_params, verbose)
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/sorters/kilosort2/kilosort2.py", line 130, in _setup_recording
    positions = np.array(recording.get_channel_locations())
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/core/baserecording.py", line 462, in get_channel_locations
    raise Exception('There are no channel locations')
Exception: There are no channel locations

mmm weird! We'll send you an email to organize a call so we can fix it once and for all :)

I'm guessing this error stack is from the update to spikeinterface 0.94 -- has this function call been moved/renamed?
SpikeGLXRecordingExtractor: 384 channels - 1 segments - 30.0kHz - 7376.323s
Sampling frequency: 30000.0
Traceback (most recent call last):
  File "/home/venditto/github/brody-lab-to-nwb/scripts/spikeglx_spikeinterface_pipeline_new.py", line 57, in <module>
    peaks = scp.detect_peaks(recording_processed, **job_kwargs)
AttributeError: module 'spikeinterface.sortingcomponents' has no attribute 'detect_peaks'

So the good news is I haven't run into any new errors; the not as good news is the job is timing out after 24 hours. I feel like it shouldn't take that long because running ks2 on our local analysis computer only takes ~2 hours for a single probe/session. I've attached the .out file so you can see where it got interrupted: slurm-8665194.out.txt. It looks like ks2 finished running, but it timed out during the post-processing. My guess is I'm not setting the parallelization parameters correctly on the cluster, but I'll admit that I'm not really sure how to properly set them. Do you have a better intuition on how to set up the job? For reference, the cluster I'm using is TigerGPU, whose specs are listed here: https://researchcomputing.princeton.edu/systems/tiger
Here are the parameters I'm setting for the Slurm job scheduler:
#SBATCH --nodes=1                             # node count
#SBATCH --ntasks=1                            # total number of tasks across all nodes
#SBATCH --cpus-per-task=7                     # cpu-cores per task (>1 if multi-threaded tasks)
#SBATCH --mem-per-cpu=16G                     # memory per cpu-core (4G is default)
#SBATCH --gres=gpu:1                          # number of gpus per node
#SBATCH --time=24:00:00                       # total run time limit (HH:MM:SS)

The SpikeInterface parallelization parameters are set at:
job_kwargs = {"n_jobs": 4, "chunk_size": 30000, "progress_bar": True}

The kilosort2 parameters are:
sorter_params = dict(
    kilosort2=dict(car=False, n_jobs_bin=4, total_memory="4G"))

I did try running once with n_jobs_bin=7 and total_memory="16G" for the kilosort2 parameters (and everything else the same as above), but that job failed with an out of memory error

@alejoe91 @CodyCBakerPhD Based on the slurm output text file Sarah Jo attached, it looks like ks2 actually did finish in a reasonable amount of time (~4000s or ~1hr) which implies the SpikeInterface stuff is what took most of the time of the job, before it timed out during "Extract Waveforms". Any tips for removing unnecessary computation steps or otherwise speeding things up? A node on our cluster can theoretically use up to 20 cores and has up to 256GB of memory across those cores, but those are not being fully used now it looks like. Each node also has up to 4 GPUs but currently SI doesn't use GPUs is that right?

Hi @sjvenditto @agbondy
Inspecting the slurm output, it seems that something weird happened at some point in the waveform extraction: up to line 7733 (~29%) the estimated time is around 1h (20:13<30:52, which is fair for 4 jobs), but then suddenly everything seems to slow down and ET quickly increases to over 30h.. I suspect this could be due to filling up the disk space of the node. Does it make any sense? Can you control the disk space allocated to a node? Note that waveform extraction, if you have many units on NP probes, creates quite large files..
Anyways, as a first try, I would start by using #SBATCH --cpus-per-task=20, job_kwargs = {"n_jobs": 20, "chunk_size": 30000, "progress_bar": True} and sorter_params = dict(kilosort2=dict(car=False, n_jobs_bin=20, total_memory="500M")) (in general smaller chunks help with parallelization.
SI currently doesn't use GPUs, but we'll try little by little to provide GO=PU backends for the main heavy functions.
@sjvenditto let me know if that helps and thank you for sharing the ouput!

Sorry for the hiatus --
It looks like the timeout was due to the extracted waveforms being saved to my local directory on TigerGPU instead of to scratch, so my local directory ran out of space. I changed the save location of the waveforms to the scratch space and it ran in a reasonable amount of time (~30 minutes), but now I've run into this error:
/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/numpy/lib/function_base.py:380: RuntimeWarning: Mean of empty slice.
  avg = a.mean(axis)
/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/numpy/core/_methods.py:181: RuntimeWarning: invalid value encountered in true_divide
  ret = um.true_divide(
/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/spikeinterface/toolkit/qualitymetrics/misc_metrics.py:241: RuntimeWarning: invalid value encountered in true_divide
  violation_rate = num_violations / violation_time
Traceback (most recent call last):
  File "/home/venditto/github/brody-lab-to-nwb/scripts/spikeglx_spikeinterface_pipeline_new.py", line 192, in <module>
    write_sorting(
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/nwb_conversion_tools/tools/spikeinterface/spikeinterface.py", line 1240, in write_sorting
    nwbfile = make_nwbfile_from_metadata(metadata=metadata)
  File "/home/venditto/.conda/envs/si_new/lib/python3.9/site-packages/nwb_conversion_tools/tools/nwb_helpers.py", line 55, in make_nwbfile_from_metadata
    assert "session_start_time" in nwbfile_kwargs, (
AssertionError: 'session_start_time' was not found in metadata['NWBFile']! Please add the correct start time of the session in ISO8601 format (%Y-%m-%dT%H:%M:%S) to this key of the metadata.

Aha! We are now out of SpikeInterface-land and into NWB Conversion Tools (NCT)-land, something I can help with;
right around the time we finished this project we were making session_start_time a required property of write_sorting and other arguments. I thought we had pinned to the previous version, but I suppose if you're seeing this error that is not the case.
If you want to write a stand-alone file with the spike sorted data, simply pass a field metadata=dict(NWBFile=dict(session_start_time=some_datetime_object)) into write_sorting and this should allow it to be specified.
to make a datetime object, simply call
from datetime import datetime

some_datetime_object = datetime(year, month, day, hour ,minute, second)

Now, if adding this to an existing NWBFile to which the raw SpikeGLX recording has been written, this field will be set automatically since it's a part of the SpikeGLX header file.

Wonderful to hear!
@alejoe91 This one is back to you - perhaps that mode option has changed behavior since the last time I messed with it (keep used to auto-load results as I recall)

@sjvenditto Yeahhh!!
So I just looked at the notebook again, and in the cell before the run_sorters we remove the working_dir no matter what! You can remove that part (if working_folder.is_dir(): shutil.rmtree(working_folder)) and then the keep should do its job (unless the folder is deleted somewhere else...).
In order to reload the pre-computed waveform extractor, you can use the load_if_exists=True argument here:
we = si.extract_waveforms(recording_processed, sorting, folder=waveform_folder, load_if_exists=True,
                                            max_spikes_per_unit=500, **job_kwargs)

Discussed with @sjvenditto and issue can now be closed.
@sjvenditto If you ever have any other trouble with this or related operations please feel free to raise a new issue


Create Bagels Game + Documentation

A deductive logic game where you must guess a number based on clues.
assign me under iwoc 24

Done 👍

sir please add label and i have created a pr
please review @TusharAMD


Setting timezone (again)

Hello,
Flask-Admin seems to show all timestamp fields as if they were UTC (which is also set in the locale of the server). However, I would like to show the dates in GMT+2, which is the timezone of all users of my app - without having to reconfigure the server it is running on.
I found #1246, which is now closed, referring to #772, which in fact discusses a different issue, and I have to concur with the last poster, that the solution there doesn't work - nothing at all changes in my forms if I override the widgets like described.
I also tried using Flask-BabelEx, setting the locale to 'cs' using @babel.localeselector, which successfully changed the form labels, but not the timezone - and changing the timezone directly, using @babel.timezoneselector, which also had no effect.
Is there a way to set Flask-Admin's timezone?

Does anyone have an update or workaround on this?

Does anyone have an update or workaround on this?

@hmanicka posted a gist  2 years ago, but I would rather let the client do at least the conversion to local time. My approach has 2 advantages: bringing some load to the client rather than the server, and automatically getting the user's timezone. Unfortunately, I am not able to make it work with datepicker. This is not an issue to me, as nowadays browser's UI for datetime-local is excellent.
templates/admin/edit.html
{% extends 'admin/model/edit.html' %}
{% block tail %}
    <script>
        fetch('/set_timezone', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify(Intl.DateTimeFormat().resolvedOptions().timeZone)
        })

        function convertToUserLocalTime() {
            const inputs = document.querySelectorAll('input[data-date-format]');
            const userTimezone = Intl.DateTimeFormat().resolvedOptions().timeZone;
            inputs.forEach(input => {
                const localizedTime = new Date(input.getAttribute("value") + "Z");
                const formattedDate = `${localizedTime.getFullYear()}-${(localizedTime.getMonth() + 1)
                    .toString()
                    .padStart(2, '0')}-${localizedTime.getDate().toString().padStart(2, '0')}T${localizedTime
                    .getHours()
                    .toString()
                    .padStart(2, '0')}:${localizedTime.getMinutes().toString().padStart(2, '0')}`;
                input.value = formattedDate;
            });
        }

        convertToUserLocalTime();
    </script>
{% endblock %}

routes.py
app.route('/set_timezone', methods=['POST'])
def set_timezone():
    current_user.timezone = request.get_json()

admin.py
class AdminView(ModelView):
    edit_template = 'admin/edit.html'  # localize datetime

    date_args_format = dict(format='%Y-%m-%dT%H:%M')
    date_widget_args_format = {
        'data-date-format': '%Y-%m-%dT%H:%M',
        'data-role': 'none',
        'type': 'datetime-local',
    }

    form_args = dict(
        datefield1=date_args_format,
        datefield2=date_args_format,
    )
    form_widget_args = dict(
        datefield1=date_widget_args_format,
        datefield2=date_widget_args_format,
    )

    def validate_form(self, form, view) -> bool:
        for field in form:
            if field.type == 'DateTimeField':
                if field.data is not None:
                    try:
                        field_data = field.data.replace(tzinfo=ZoneInfo(current_user.timezone))
                        # convert to UTC
                        field.data = field_data.astimezone(ZoneInfo("UTC"))
                    except ZoneInfoNotFoundError as e:
                        current_app.logger.exception(e)
                        flash(current_user.timezone + _(
                            ' is not an accepted timezone. Please use a different timezone.'), 'danger')
                        return False
        return super(view, self).validate_form(form)


vagrant up?

I am new to vagrant and packer so forgive me. I  successfully did all four steps mentioned in the README.md.

git clone https://github.com/StefanScherer/packer-windows
cd packer-windows
packer build --only=vmware-iso windows_2016_docker.json
vagrant box add windows_2016_docker windows_2016_docker_vmware.box

Now I try 'vagrant up' and get
$ vagrant up
A Vagrant environment or target machine is required to run this
command. Run `vagrant init` to create a new Vagrant environment. Or,
get an ID of a target machine from `vagrant global-status` to run
this command on. A final option is to change to a directory with a
Vagrantfile and to try again.
$

I must have missed a step. Is there any possibility of getting some high-level explanation for the last two steps and what I need to do to be able to run 'vagrant up' successfully.
Thank you.

You have to clone/download this github repo and change the current dir into it so Vgrant can find the Vagrantfile and spin up the VM.

Before vagrant up you have to go to the docker-windows-box folder where the Vagrantfile is. If you typed the packer build steps then a cd .. should do that.

I'll close this issue. If you still need help feel free to reopen it.


Image processing unit tests

unit tests all pass locally but not on travis and it's driving me insaneeeee

Uh oh, I think I merged in some conflicts into my branch from @AznStevy

How so?

Not sure. I might have merged an older version of your deveop-webserver branch or something and now I can't merge my unit_testing branch with fake_master because of conflicts. It's not a big deal, I know which of my files I need, so we could technically just abandon this branch if we have to.


[Processing Request]: Request for 3 MGRS Tiles for 2022 and 2023

Venue
PST
Product
DSWx-S1
SAS Version
No response
SDS Version
No response
Input Data
The CSV contains the 455 requested input granules and their associated MGRS tiles.
jwjones_request.csv
Share Results

[ ] Google Earth Engine (GEE)
[ ] NASA Distributed Active Archive Centers (DAAC) User Acceptance Testing (UAT)

Additional Notes
Keep us posted with timeframe of this request. Thanks!

DSWx-S1 products were made for all MGRS tilesets with any coverage from the RTC products derived from the SLCs given above.
Let me know if you'd like me to compile any output file sets for review.

Hey @niarenaw - I am seeing far fewer products than I expected. For the 3 tiles I requested, there are only 55 products. In the CSV, I provided there are 234 unique dates across 2022 and 2023. 234 is a lower bound because two of the tiles share common tracks. Put slightly differently, I only see products spanning June 2022 to October 2023 and the SLCs go beyond that.
Can you please explain. Thanks.

@cmarshak Totally jumped the gun and triggered DSWx-S1 processing before all RTCs were delivered. Working on resolving now.

It might be good to delete the 55 products - I see some products that are missing the "interior" of the product. It will also be easier to search if the only products in a given tile are those you just generated :)
Thanks!

Sure thing -- I'll clean up those products.

Adding a note this is the request for the DSWx-S1 Inundated Vegetation validation sites

Outputs from second processing campaing:
request-43-outputs-second-processing-campaign.txt


added navabr workflow

is there a way that i can preview the navbar?

https://turinglang.org/AbstractMCMC.jl/

Thanks @jeetsuthar


error upon integration setup

ERROR (MainThread) [homeassistant.config_entries] Error occurred loading flow for integration powertag_gateway: cannot import name 'make_byte_string' from 'pymodbus.utilities' (/usr/local/lib/python3.11/site-packages/pymodbus/utilities.py)
Home Assistant 2023.10.1
OS: Ubuntu 22.04
Docker
This happens when using latest stable version from HACS, and also the beta 1.0.5

beta 1.0.5

You mean 0.1.5, right?
Can you please check your logs, search for powertag_gateway and copy me the error that you will be seeing there? It will contain more info that will help me find the cause.
Other integrations have reported this problem as well:
https://github.com/binsentsu/home-assistant-solaredge-modbus/discussions/175
https://github.com/wills106/homeassistant-solax-modbus/issues/464
I think that your Ubuntu has a bad version of pymodbus installed. This is just guesswork, but please try this:
pip install --upgrade --force-reinstall pymodbus

Yeah, sorry I meant 0.1.5 ;)
I tried to force upgrade the pymodbus package, but that didn't affect the outcome.
I have also included:
logger:
logs:
custom_components.powertag_gateway: debug
But i don't get anymore debug information than I already provided above :/

Thanks! Updating pymodbus inside the docker container made some progress. But I'm still not able to complete the setup.

log:
INFO (SyncWorker_24) [custom_components.powertag_gateway.config_flow] Attempting to discover EnergyTag Gateway  INFO (MainThread) [custom_components.powertag_gateway.config_flow] Found 1 candidates...  INFO (MainThread) [custom_components.powertag_gateway.schneider_modbus] Connecting Modbus TCP to <IP_ADDRESS>:502   File "/config/custom_components/powertag_gateway/config_flow.py", line 113, in async_step_user   File "/config/custom_components/powertag_gateway/config_flow.py", line 142, in async_step_device   File "/config/custom_components/powertag_gateway/config_flow.py", line 82, in async_discovery   File "/config/custom_components/powertag_gateway/config_flow.py", line 47, in __init__   File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 156, in __init__   File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 161, in find_synthentic_table_slave_id   File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 654, in __read_int_16   File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 629, in decoder

and trying with manual connect:

log:
INFO (MainThread) [custom_components.powertag_gateway.schneider_modbus] Connecting Modbus TCP to <IP_ADDRESS>:502

I already have the powertags integrated in HA using pure modbus config. Could that be conflicting?

Hmm it just fails to connect, there's little we can do about that.
If it already works with modbus directly, that means the IP and port are configured correctly. You do everything correctly.
The existing config shouldn't conflict, but it's worth a shot to comment out that config. There's nothing else you can try I'm afraid.

okay, but thank you very much for your time and help! I hope i can make this work some day. Would really like to use this integration instead of the yaml based modbus one..
Another thing: when i reboot my docker container, it seems like the pymodbus package isn't persistent. Which mean i would need to update the pymodbus library on every restart of the container. any way to make this persistent? Or would i have to compile my own image which includes the newer pymodbus package then?

Try upgrading the docker image that you use, I'm assuming it's an older version in that case.

Can you please show me your Dockerfile?

well, it actually some time since i've used dockerfile. I administrate the container from portainer webui. So I actually just recreate the existing container to pull the latest image. Could I just share some screendumps from the portainer webui for you, If you specify what options you are curious about?

posting some info about the GW, in case my specific device isn't supported:

Ah I think I got it. Hold on.

Ok try with v0.1.8. That should revolve the first error permanently.
No clue on your connection problem though.

I've updated to v0.1.8 and restarted HA (not container). these are the logs:
2023-10-13 11:33:04.394 INFO (SyncWorker_45) [custom_components.powertag_gateway.config_flow] Attempting to discover EnergyTag Gateway
2023-10-13 11:33:07.498 INFO (MainThread) [custom_components.powertag_gateway.config_flow] Found 2 candidates...
2023-10-13 11:33:07.512 INFO (MainThread) [custom_components.powertag_gateway.schneider_modbus] Connecting Modbus TCP to <IP_ADDRESS>:502
File "/config/custom_components/powertag_gateway/config_flow.py", line 113, in async_step_user
File "/config/custom_components/powertag_gateway/config_flow.py", line 142, in async_step_device
File "/config/custom_components/powertag_gateway/config_flow.py", line 82, in async_discovery
File "/config/custom_components/powertag_gateway/config_flow.py", line 47, in init
File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 156, in init
File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 161, in find_synthentic_table_slave_id
File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 654, in __read_int_16
File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 629, in decoder

Actually, its acting the same as earlier. Auto-discovery finds GW, but throwing "unknown error occurd" on connecting. Manual connection states "cannot connect", and the log just shows:
2023-10-13 11:37:13.965 INFO (MainThread) [custom_components.powertag_gateway.schneider_modbus] Connecting Modbus TCP to <IP_ADDRESS>:502

is it possible to add some more debugging to the code, so we can see whats going on in the connection phase?

I've updated to v0.1.8 and restarted HA (not container). these are the logs:
2023-10-13 11:33:04.394 INFO (SyncWorker_45) [custom_components.powertag_gateway.config_flow] Attempting to discover EnergyTag Gateway 2023-10-13 11:33:07.498 INFO (MainThread) [custom_components.powertag_gateway.config_flow] Found 2 candidates... 2023-10-13 11:33:07.512 INFO (MainThread) [custom_components.powertag_gateway.schneider_modbus] Connecting Modbus TCP to <IP_ADDRESS>:502 File "/config/custom_components/powertag_gateway/config_flow.py", line 113, in async_step_user File "/config/custom_components/powertag_gateway/config_flow.py", line 142, in async_step_device File "/config/custom_components/powertag_gateway/config_flow.py", line 82, in async_discovery File "/config/custom_components/powertag_gateway/config_flow.py", line 47, in init File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 156, in init File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 161, in find_synthentic_table_slave_id File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 654, in __read_int_16 File "/config/custom_components/powertag_gateway/schneider_modbus.py", line 629, in decoder

Resolved in v0.1.9, sorry about that.

is it possible to add some more debugging to the code, so we can see whats going on in the connection phase?

cannot_connect means it's network related. There's little I can do from my end.
You can try going inside your docker container again and ping <IP_ADDRESS>, or telnet <IP_ADDRESS> 502 to see if the port is open.

Problem solved! Something you did in the latest release fixed it! :D Thank you so much!! :)


SMTP modifications

Added from, replyTo, and displayName properties to SMTPEmailService
Removed from, replyTo from EmailMessage
Added displayName property

Sorry for the late drive-by here but this PR/design doesn't feel right and I'd like to know a little more about it. Why are 'from' and 'replyTo' not properties on EmailMessage? From a modelling standpoint, I don't see why they shouldn't be.
I understand that the SMTPEmailService might not have permissions to send from any email address but I don't think the SMTPEmailService needs to care about that. Telling it to send an email from an email address it can't send from is more an issue with the caller than the SMTPEmailService.
There's a couple of issues with the sendEmailFromTemplate API method. According to the API, first argument is the user sending the email but the SMTPEmailService isn't respecting that because it fills in its own from address. If EmailMessage had a 'from' address then this first argument would be unnecessary and could be removed to simplify the API.
This stuff is hiding a bug in the ResetPasswordTokenService:
https://github.com/eatonk/foam2/blob/cee3f16077a625b684c675234817f72e85745630/src/foam/nanos/auth/resetPassword/ResetPasswordTokenService.js#L88
The first argument of the call to sendEmailFromTemplate is the user that initiated the password reset which is not who we'd want to be sending the email from.
Am I missing something here or should we file a bunch of cleanup bugs to address the issue? The bugs I'd suggest are:

Add from/replyTo to EmailMessage and remove them from SMTPEmailService
Remove the user argument from sendEmailFromTemplate
Fix up any calls to sendEmailFromTemplate that are relying on the incorrect behaviour.


Raise exception for non existent datasets

Small improvement to get a better error message in case a dataset_name is used, which is not supported.

Thanks again for a great addition! LGTM :+1:


Upload regression as CSV

Part of: #3981
Description
This pull request includes changes to the .github/workflows/upload-perf-results.yml file to enhance the performance results upload process by adding steps to download and run a regression script, and upload the regression results.
Enhancements to performance results upload process:

Added a step to download and run a regression script from a specified URL and save the results to a CSV file.
Added a step to upload the regression results CSV file as an artifact using the actions/upload-artifact action.

Testing
CI/CD
Documentation
No.
Installation
No.

Need to parameterize the script to select the source of the metrics


K8SSAND-1763 ⁃ Operator should not allow rack name change of a live cluster

What is missing?
Changing the rack name of a live cluster (and thus changing the NetworkTopology) is unsupported.
The operator should reflect that and explicitely block this operation.
Instead, it tries to carry out this change and new Cassandra replicas display such errors:
INFO  [main] 2022-09-03 17:22:43,469 NetworkTopologyStrategy.java:88 - Configured datacenter replicas are dc1:rf(3)
ERROR [main] 2022-09-03 17:22:43,472 CassandraDaemon.java:911 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Token allocation failed: the number of racks 2 in datacenter dc1 is lower than its replication factor 3.

Why do we need it?
Protect from accidental changes
Make the operator safer to use
Environment

K8ssandra Operator version: 1.2.0

┆Issue is synchronized with this Jira Task by Unito
┆friendlyId: K8SSAND-1763
┆priority: Medium

I agree that rack changes shouldn't be allowed in live datacenters. Any such change should be done by migrating to a new datacenter that uses the required rack assignments. Even if it were done in a way that Cassandra accepts, this could lead to data loss (or at least reduce the consistency guarantees).
We should treat this the same way we do with attempts to change the number of tokens, prevent it both in the validating webhook and in the K8ssandraCluster reconcile.
The same check should probably be done in cass-operator too.

cass-operator already prevents renaming racks:
https://github.com/k8ssandra/cass-operator/blob/master/apis/cassandra/v1beta1/cassandradatacenter_webhook.go#L189
There's an open ticket to allow removing a rack, which is not supported at the moment. Adding racks is a supported feature.

cool, all we need is to add that check in k8ssandra-operator as well then.

I talked with @olim7t and we agree that there are other situations where we will want similar, duplicate validation checks. He is going to investigate to see if we can minimize that duplication.

After more discussion with the team, we are not convinced anymore that duplicating the validation logic is a good solution. It's not just rack names, there are many more things being checked by cass-operator. The two controllers operate on different structs so it is hard to minimize the duplication.
With the current version of the code, any validation error in cass-operator will be reported in the Status.Error field of the K8ssandraCluster, which makes it pretty visible to the user.

@maximethebault, looking at the original ticket description, it doesn't seem like what you've done is a rack name change but rather removing a rack.
The error message you're seeing suggests that:
ERROR [main] 2022-09-03 17:22:43,472 CassandraDaemon.java:911 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Token allocation failed: the number of racks 2 in datacenter dc1 is lower than its replication factor 3.

Could you confirm?

@adejanovski Sorry for the late reply, missed the notification.
I had the error during a rack name change. I posted more details here: https://discord.com/channels/836217371453685760/836217371453685763/1015677950869381230
Note that in the discord messages, I added a rack configuration to a cluster that had none defined (I guess in this case a rack is created implicitly), but I can reproduce the exact same issue by changing the name of an explicitly defined rack.

This is no longer possible in the latest versions.
I tried to follow the steps you described on Discord and this is what I get:
2022-12-19T07:22:22.837Z	ERROR	Reconciler error	{"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"cluster","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "cluster", "reconcileID": "a7d279cb-b1b1-4ac1-8c25-8c15a065ed9c", "error": "admission webhook \"vcassandradatacenter.kb.io\" denied the request: CassandraDatacenter write rejected, attempted to change rack name from 'default' to 'r1'"}

I'll go ahead and close this issue.
Feel free to reopen if you can reproduce this in the latest release.

For reasons completely unrelated to K8ssandra, I'm unfortunately stuck at v1.2.1 atm, but I'll make sure to test it out as soon as I can.
Merci!


Typo Updated for kube-hunter docs to sound more meaningful

Description
Please include a summary of the change and which issue is fixed. Also include relevant motivation and context. List any dependencies that are required for this change.
Contribution Guidelines
Please Read through the Contribution Guidelines.
Fixed Issues
Please mention any issues fixed in the PR by referencing it properly in the commit message.
As per the convention, use appropriate keywords such as fixes, closes, resolves to automatically refer the issue.
Please consult official github documentation for details.
Fixes #(issue)
"BEFORE" and "AFTER" output
To verify that the change works as desired, please include an output of terminal before and after the changes under headings "BEFORE" and "AFTER".
BEFORE
Any Terminal Output Before Changes.
AFTER
Any Terminal Output Before Changes.
Contribution checklist

[x] I have read the Contributing Guidelines.
[x] The commits refer to an active issue in the repository.
[ ] I have added automated testing to cover this case.

Notes
Please mention if you have not checked any of the above boxes.

Hi,
Thanks for your time in making this contribution.
However this adds no value to the README, so we will not merge it.
Don't let this discourage you though, I'm sure that with some time and motivation you can find something you can fix/add value with.


Add final modifier to several public class fields

Makes robust and improve performance.
Add the final modifier to the several public class fields, and change scope to the package private.

Coverage decreased (-0.006%) to 95.45% when pulling 5ffc0d5fa63c249ccffac4d75584e3c2d2392330 on t-tsutsumi:master into 082fce76eb7204b30931f9ba5ee8919ab506b2ce on tfahub:master.


Literal::Array#-

@christopher-b can you comment on this issue so I can assign it to you?

👋🏻

Implemented in ba4988b03a842e5cd9bfe04833677e5c5ea82e63, as part of my omnibus PR


Update to Minecraft 1.20.5-6

Please. ♥

Hoping for it too

https://github.com/Roundaround/mc-fabric-roundalib/issues/1

Now hoping just for 1.20.6 since most mods already transitioned to it.
@Roundaround also congratulations :)

Roundaround/mc-fabric-roundalib#1

congratulations on your daughter!


Upgrade to controller-runtime 0.8.3

Update controller-runtime to v0.8.3 (also updates various Kubernetes dependencies)
controller-runtime introduced a new client.Object interface which is a superset of runtime.Object. Use that where appropriate.
controller-runtime changed the Reconcile interface to include a context.Context
Update RBAC to allow use of the new coordination.k8s.io API in Kubernetes

Fixes #7

/lgtm
/approve


promtool test with /dev/stdin fails for multiple test cases

What did you do?
I use promtool test with the following rules and tests:
rules
# prometheus-rules.yml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
spec:
  groups:
    - name: myalerts
      rules:
        - alert: ErrorInLog
          expr: changes(log_messages_total{log_level="ERROR"}[1m]) > 0
          annotations:
            summary: 'New error(s) appeared in the log of container {{ $labels.container }}'
        - alert: NoNewMessages
          expr: sum(increase(messages_sent_total[3m])) == 0
          annotations:
            summary: 'My component did not send any messages for 3 minutes'

tests
# test-alerts.yml
rule_files:
  - /dev/stdin

tests:
  - name: 'Should trigger if no messages were sent for more than 3 minutes'
    interval: 1m
    input_series:
      - series: 'messages_sent_total'
        values: '0 0 0 0'
    alert_rule_test:
      - eval_time: 4m
        alertname: NoNewMessages
        exp_alerts:
          - exp_annotations:
              summary: 'My component did not send any messages for 3 minutes'
  - name: 'should trigger in case an error is detected'
    interval: 1m
    input_series:
      - series: 'log_messages_total{container="mycontainer", log_level="ERROR"}'
        values: '0 1 2'
    alert_rule_test:
      - eval_time: 2m
        alertname: ErrorInLog
        exp_alerts:
          - exp_labels:
              log_level: "ERROR"
              container: "mycontainer"
            exp_annotations:
              summary: 'New error(s) appeared in the log of container mycontainer'

It seems that there is a bug with the way promtool reads from /dev/stdin compared to executing the tests with a static file.
What did you expect to see?
The same tests with a static input file (alerts.yaml) pass:
 yq .spec prometheus-rules.yml > alerts.yaml && promtool test  rules test-alerts.yml && rm alerts.yaml
Unit Testing:  test-alerts.yml
  SUCCESS

What did you see instead? Under which circumstances?
The tests fail for /dev/stdin:
 yq .spec prometheus-rules.yml | promtool test  rules test-alerts.yml
Unit Testing:  test-alerts.yml
  FAILED:
    name: should trigger in case an error is detected,
    alertname: ErrorInLog, time: 2m,
        exp:[
            0:
              Labels:{alertname="ErrorInLog", container="mycontainer", log_level="ERROR"}
              Annotations:{summary="New error(s) appeared in the log of container mycontainer"}
            ],
        got:[]

System information
macOS 14.4
Prometheus version
promtool, version 2.50.1 (branch: non-git, revision: non-git)
  build user<EMAIL_ADDRESS>  build date:       20240226-11:31:35
  go version:       go1.22.0
  platform:         darwin/arm64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file
No response
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response

The key here is multiple, each test reads from /dev/stidn. I think it would be useful to support use cases like your yq one by buffering to a temp file so it works.

Also you don't see a read error because further reads from /dev/stdin are reading from the open FD, which has reached the end, so each read just gets an EOF. Prometheus in general accepts empty rule files, but it's possible to specially handle this in promtool tests so stdin/pipes get errors on an empty read, I've put something in the PR to deal with that too.


Peer dependencies seem to be rendered only for root packages

While devDependencies is indeed a root-only concept, peerDependencies is not (the peer dependencies of your dependencies must be satisfied). So to me, the peer dependencies deeper in the tree should be rendered as links too (the dependency resolution should make them link to the package depended on by the parent)

they should not be ghosted, as they are normal dependencies

I disagree.  If I do npm install autoprefixer, I don't get the peer dependencies.  It is incumbent upon me to explicitly add them. I have to explicitely type npm install postcss so that postcss is added to my normal dependencies list.  If I do that then, sure, we should show postcss as a normal module (but probably still shade the dependency line to it).
But if I haven't explicitly installed postcss, then neither it nor any of it's sub-dependencies will exist, and we shouldn't indicate anything to the contrary.

@broofa I mean that they should not  be ghosted when we link to the dependency provided by the parent, as postcss is not a ghost in the tree then.
However, the link itself should still have a style indicating a peer dependency (dashed linked in your proposal, green link in the existing code)


Remove skip_metadata_api_check = true from TF AWS Provider

closes https://github.com/DataDog/stratus-red-team/issues/367.
TL;DR: AWS Providers initialised with skip_metadata_api_check = true prevent Stratus techniques executing within AWS contexts from discovering instance profile credentials. This change will allow the terraform provider to use the instance-profile assigned to the EC2 within which Stratus is running as it won't skip checking the Instance Metadata for those credentials.

Looks good to me - just fixed a small Terraform formatting issue. Thank you for the contribution!


Testnetwork needs to be adjusted to work with V2 and V3

So currently we need to get the local testnetwork working. This means that we should be able to run medium-network.toml with zombienet, then run yarn start:zombienet-post-script which should succesfully create a test asset on our local system parachain.
The current issue is when the xcm transfer is sent that there is a Barrier xcm error on the system parachain end.
@bee344 I would suggest reading the zombienet section of the README.md, and getting the network working locally. Then try running the script and monitoring the messages sent through polkadot-js.


Custom covers stop loading

Steps to reproduce

Add a web novel (from any source) to your library
Use 'Edit Cover' to change the cover.
After a while, for some unknown reason, the cover stops getting loaded and shows as empty.

Expected behavior
It should keep the custom cover I have set.
Actual behavior
The cover is missing (like there is no cover).

LNReader version
1.1.19
Android version
Android 11
Device
Mi 8
Other details
Although I use v1.1.19, I didn't find it in the changelog for v2.0.0, so I believe that it hasn't been fixed.
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] If this is an issue with a source, I should be opening an issue in the sources repository.
[X] I have updated the app to version 2.0.0.
[X] I will fill out all of the requested information in this form.

For me covers from novelbin cant be fetched too. Only ScribbleHub Covers will load.

I dont know how you checked the changelog, but I believe it is fixed in v2 app

I dont know how you checked the changelog, but I believe it is fixed in v2 app

I believe this also happens with local epub, sometimes it doesn't load the covers, maybe it's related?

send vol 14 epub


Drive is including Accept: text/vnd.turbo-stream.html on visits that are not inside of a Frame

We are working on transitioning a JS-heavy filtered list to Turbo Frames + Streams, and something we've run into seems like it might be a bug in Turbo.
We have a DealsController#index action that is one of the primary parts of our application, with a link in our main navigation to it like this:
Deals

We've got Turbo Drive set up for the application, so with no introduction of Frames/Streams, this works perfectly, it updates the  through Drive as expected. Then we started introducing <turbo-frame>s to support things like infinite scrolling of "deals", with our index.html.erb looking something like this:
<%= turbo_frame_tag 'deals', target: '_top' do %>
  <%= render @deals %>
<% end %>

<%= turbo_frame_tag 'deals-next-page', src: url_for(page: @page + 1), loading: 'lazy' %>

And an index.turbo_stream.erb that looks something like this:
<%= turbo_stream.append 'deals' do %>
  <%= render partial: @deals, formats: %i[html] %>
<% end %>

<%= turbo_stream.replace 'deals-next-page' do %>
  <%= turbo_frame_tag 'deals-next-page', src: url_for(page: @page + 1), loading: 'lazy' %>
<% end %>

However, when we introduce index.turbo_stream.erb, our navigation link (which is outside of any <turbo-frame> and should behave just as a simple Drive request) no longer works as expected. Instead of getting the full HTML response from the server and replacing the , it receives index.turbo_stream.erb, which makes for all sorts of weird issues when clicking that navigation link from other pages (i.e. from our home page, where there is no <turbo-frame id: 'deals'> element on the page).
After looking into this a bit more, I'm pretty sure it's because Turbo Drive is including the Accept: text/vnd.turbo-stream.html header with all of its requests, which is causing my controller to respond with that format if it's available.
When clicking a link (or submitting a form) inside of a <turbo-frame> (without a target='_top'), that makes perfect sense to me, but I'm surprised it does this when clicking a link anywhere else on the page. In that case, I'd expect Turbo Drive to do its thing, and simply fetch the standard HTML response from the server, without the text/vnd.turbo-stream.html content type included in the Accept header.
Am I misunderstanding something basic about Turbo, or is this something that should be fixed?

Thank you for writing this up, @agrobbin. Would it be resolved by https://github.com/hotwired/turbo/pull/52?

Thank you for writing this up, @agrobbin. Would it be resolved by https://github.com/hotwired/turbo/pull/52?

My pleasure @seanpdoyle, thanks for the quick response!
It looks like #52 would fix this, though I'm curious, how would that play with GET requests initiated within a <turbo-frame>? Wouldn't those then lose the Accept: text/vnd.turbo-stream.html header? Or is that accounted for in some other way?

My pleasure @seanpdoyle, thanks for the quick response!
It looks like #52 would fix this, though I'm curious, how would that play with GET requests initiated within a <turbo-frame>? Wouldn't those then lose the Accept: text/vnd.turbo-stream.html header? Or is that accounted for in some other way?

Oh wait, I think I see, it sends the Turbo-Frame header, that makes sense!

Oh wait, I think I see, it sends the Turbo-Frame header, that makes sense!

Thanks for the quick resolution here @seanpdoyle and @sstephenson! Looking forward to seeing this fix released in the next beta.

Thanks for the quick resolution here @seanpdoyle and @sstephenson! Looking forward to seeing this fix released in the next beta.

@agrobbin after the change introduced in #52 , does your infinite scroll approach not break?
The initial page load would render index.html.erb with something like:
<%= turbo_frame_tag 'deals', target: '_top' do %>
  <%= render @deals %>
<% end %>

<%= turbo_frame_tag 'deals-next-page', src: url_for(page: @page + 1), loading: 'lazy' %>

just fine.
But the Fetch initiated by <%= turbo_frame_tag 'deals-next-page', src: url_for(page: @page + 1), loading: 'lazy' %> no longer has the Accept: text/vnd.turbo-stream.html header, and so loads index.html.erb instead of the desired index.turbo_frame.erb.  Dose it not?

@spinosa yep this is definitely the case. After the changes in #52, I went a different route, with a separate search action that handles the different rendering for page 2+.
What should make this a lot simpler is the work being done in #146!

@agrobbin I ended up with a similar refactoring.
The paged index route implemented with Turbo Streams and overloaded views is elegant; 🙏 #146

@agrobbin You can use on view link with index_path(format: :html) then Rails controller will process request as HTML


Update compare_cmaps to print details about cmap differences.

Details will list characters by block, and show for removed
characters the remaining locations of those characters.

LGTM


Custom example

@kalwalt I'm trying to isolate only what the NFT example needs, but I'm getting a 404 from the artoolkit worker.  The example is from V3 branch.
The error occurs after  Pattern ratio size set to 0.500000.
onError | @ | artoolkit.worker.js:49
(anonymous) | @ | artoolkit.min.js:1
oReq.onload | @ | artoolkit.min.js:1
load (async)
ajax | @ | artoolkit.min.js:1
loadCamera | @ | artoolkit.min.js:1
ARCameraParam.load | @ | artoolkit.min.js:1
ARCameraParam | @ | artoolkit.min.js:1
load | @ | artoolkit.worker.js:53
self.onmessage | @ | artoolkit.worker.js:8
Project structure:

vendor

jsartoolkit

...

src

ar.min
three.min
pinball files
camera.para.dat
gltf loader

flamingo.gb
nft.html

Got it to work.
It was the var path = '../../'; at artoolkit.worker; line 25 in the loader function.
It gets the descriptorsUrl from the markerControls and tries to make it relative from where the artoolkit.worker is.

@Carnaux maybe @nicolocarpignoli changes the path for the aframe example, i know that should be improved.

You should try both examples, aframe and Three.js for a comparison.

I hope to have a bit of time tomorrow to continue in the development of this!

Maybe we can implement an automatic relative path solver. It would get the full path to the marker and the full path to the worker and solve it.

You should try both examples, aframe and Three.js for a comparison.

I will test it!

Adding another thing:
var worker = new Worker(       THREEx.ArToolkitContext.baseURL +         "vendor/jsartoolkit5/js/artoolkit.worker.js"  );
at ar.min.js in the handleNFT function
This path might fail using Github Pages since it ignores the "vendor" directory.

I think The worker should be injected not with a path but with a Blob object, in this way we should avoid that issue. But i didn't tested yet, because we would do this in a second moment as the ratio issue was the main issue.

I would continue this discussion but it's very late here! I'm sorry... :slightly_smiling_face:


Making results more local

Im trying to get my results to be more local..
GooglePlacesSearchQuery={{
        location: "49.8951,97.1384",
        radius: "500",
        region: ".ca",
        rankby: "distance"
      }}

any help?
Thanks.

                // available options: https://developers.google.com/places/web-service/autocomplete
                key: 'my key',
                language: 'en', // language of the results
                location: `${initialLocation.latitude},${initialLocation.longitude}`,
                radius: 1000*200,
              }}

works for me


Cannot find ./upload folder generated during manual deployment

Problem
Following the instructions on the Shopify Slate page: https://shopify.github.io/slate/
Under "Connect your project with your Shopify store" in task 2, I run slate deploy -m, which results in the following error where the upload/ folder cannot be found, even though it very well exists.
...
...
[02:29:39] all files 158.29 kB
[02:29:42] Finished 'build:svg' after 7.25 s
[02:29:42] Finished 'build:assets' after 7.71 s
[02:29:42] Finished 'build:zip' after 8.56 s
[02:29:42] Starting 'compress'...
[02:29:42] mk-studio-theme.zip 133.15 kB
[02:29:42] Finished 'compress' after 291 ms
[02:29:42] Starting 'output:errors'...
[02:29:42] Finished 'output:errors' after 134 μs
[02:29:42] Finished 'zip' after 8.86 s
[02:29:42] Starting 'open:admin'...
[02:29:42] Finished 'open:admin' after 127 ms
[02:29:42] Starting 'open:zip'...
[02:29:59] 'open:zip' errored after 17 s
[02:29:59] Error: Command failed: start "" "./upload/"
The system cannot find the file ./upload/.

    at ChildProcess.exithandler (child_process.js:270:12)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at maybeClose (internal/child_process.js:897:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:208:5)
[02:29:59] 'deploy:manual' errored after 26 s
[02:29:59] Error in plugin 'run-sequence(open:zip)'
Message:
    Command failed: start "" "./upload/"
The system cannot find the file ./upload/.

Details:
    cause: Error: Command failed: start "" "./upload/"
The system cannot find the file ./upload/.

    isOperational: true
    killed: false
    code: 1
    signal: null
    cmd: start "" "./upload/"
Stack:
Error: Command failed: start "" "./upload/"
The system cannot find the file ./upload/.

    at ChildProcess.exithandler (child_process.js:270:12)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at maybeClose (internal/child_process.js:897:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:208:5)

Replication steps

Install latest Slate module (v0.12.1): npm i -g @shopify/slate
Follow the Shopify Slate guide at: https://shopify.github.io/slate/

More Information
OS: Windows 7 Pro x64
Node: v8.1.4
NPM: 5.0.3

Theme folder organization, post deploy compilation step:
- my-theme-folder
    - dist/
    - src/
    - upload/    <-- it's there, but not where deploy is expecting it I guess
    - .gitignore
    - config.yml
    - package-lock.json
    - package.json

Same error here, with these specifications:
OS: Windows 7 Home Premium SP1
Slate: 0.12.1
Node: v6.11.2
NPM: 3.10.10

Any updates? Or workarounds?

We should be exploring this soon for v0.13.0. Will keep you posted.

Slate appears to have still created the zip file in an "upload" folder.
Is it safe to continue to step 3 (upload zip file) despite the error?

I installed node.js and slate today and got the same error.
But I found the upload directory.
Can I use it though?
OS: Windows 10 Pro 10.0.15063 Build 15063
Slate: 0.12.1
Node: v6.11.4
NPM: 3.10.10

Thanks everyone for reporting this. I was able to replicate it — a fix is on the way (see #318). In the meantime, you should be able to use the zip file generated by Slate without any issue.


Rename fastai v2 to fastai

Motivation

Follow up of

https://github.com/optuna/optuna-examples/pull/256
As the PR removes fastai v1, I renamed fastai v2 to fastai.

Description of the changes

Rename fastai v2 to fastai
Add version constraint of fast ai

@eukaryo Could you review this PR?


just is now in Arch official repos

I pushed just to official Arch repos so the README should reflect that.

Awesome, thank you!


create new msa_alignment subworkflow

Merging.
To do: update-snapshots for msa_alignment, untar, pigz. Ignore conda test for msa_alignment.


C++ Warning fixes, werror on ci, C++17 compliance

What

Fixes #2903
Part of #2919
currate warning list for gcc/clang

in theory this should now catch C++

enable warnings as errors on CI, for simplicity it's always active on roundtrip tests
enable standard set of warning settings on all C++ targets
fix warnings in codegen'ed code and manually written code
fix accidental C++20 (should have been caught in theory by warnings but it didn't trigger for unknown reason)

Checklist

[x] I have read and agree to Contributor Guide and the Code of Conduct
[x] I've included a screenshot or gif (if applicable)
[x] I have tested demo.rerun.io (if applicable)

PR Build Summary
Docs preview
Examples preview

fix accidental C++20 (should have been caught in theory by warnings but it didn't trigger for unknown reason)

Try -Wc99-extensions

-Wc99-extensions sadly still didn't catch
        rr_data_row c_data_row {
            .entity_path = entity_path,
            .num_instances = static_cast<uint32_t>(num_instances),
            .num_data_cells = static_cast<uint32_t>(num_data_cells),
            .data_cells = c_data_cells.get(),
        };
        ```

but left it in anyways

Remaining warning happens to be fixed in #2959. Not duplicating it in here, waiting for that instead!


conflicting types with @types/node v12

Bug Report
I don't know if there is a solution for this since it seems to be linked with @types/node version but I'm reporting it anyway just in case.
Current Behavior
When running karma tests in an angular project using @types/node v12, I get the following conflicting types error:
Error: node_modules/rxjs/dist/types/internal/types.d.ts:17:18 - error TS2717: Subsequent property declarations must have the same type.  Property 'observable' must be of type 'symbol', but here has type 'unique symbol'.

         readonly observable: unique symbol;

  node_modules/@types/node/globals.d.ts:146:14
         readonly observable: symbol;

    'observable' was also declared here.

When switching to @types/node >=13.1.0 there is no error anymore.
Expected behavior
Error should not be present.
Environment

Runtime: Node v12
RxJS version: 7.0.0-beta.10
@types/node ^12.0.0

It's breaking change listed in https://github.com/ReactiveX/rxjs/blob/master/CHANGELOG.md#700-beta9-2020-12-07 and partially leftover issue is filed at https://github.com/ReactiveX/rxjs/issues/5919 .

It's breaking change listed in https://github.com/ReactiveX/rxjs/blob/master/CHANGELOG.md#700-beta9-2020-12-07 and partially leftover issue is filed at https://github.com/ReactiveX/rxjs/issues/5919 .

Damn, I read the changelog and missed this item. Thanks for pointing that out

Damn, I read the changelog and missed this item. Thanks for pointing that out


ppfetch: init at 0.6.0

Adding yet another system information tool, this time focusing on color, pride and portability in mind.
Description of changes
This is a fork of the system information tool pfetch. It adds LGBT pride colors options alongside other improvements.
It should work across on all platforms Nix is available on with no exception, making it useful to reside on the upstream repo.
https://github.com/yael333/ppfetch

Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ *] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Thank you so much for the relevant and in-depth explanation and clarification, it's invaluable and helps a lot with a first time contribution. I've amended the init commit and added myself as a maintainer.

Thank you, believe it should be fixed now

Result of nixpkgs-review pr 328764 run on aarch64-darwin 1

1 package failed to build:

ppfetch

Result of nixpkgs-review pr 328764 run on aarch64-darwin 1

1 package failed to build:

ppfetch

I'm really sorry, the hash was definitely off and it should be fixed now

Result of nixpkgs-review pr 328764 run on aarch64-darwin 1

1 package built:

ppfetch

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1845


Unusual UI and can't connect to steam

Closing as unrelated to this project. See https://github.com/gogcom/galaxy-integrations-python-api/issues/new/choose to find proper place to report Galaxy or plugin issues.


Convert record email correspondence flow to 1 page form

This converts the "email correspondence" flow from being a 2-page flow with a confirmation screen, to a 1-page form with no confirmation screen. This is to make it more consistent with the other flows (eg risk assessments), and to enable an editing feature to be implemented more easily (ie without sending users back through a multi-page flow).
Implementation notes
I've removed Wicked Wizard, and simplified the controller and the routes to just using the :new, :create and :show actions. A form class has been introduced, and the view has been switched to use the new ApplicationFormBuilder helpers. The actual saving of the email model has been moved into a service class.
One tricky bit is making sure that any uploaded file is saved, even if there’s a validation error on the form. This seems to be the default behaviour in Rails 5, although there’s no built-in way within Rails 5 to associate the saved file with the re-rendered form, but in Rails 6 this will no longer happen automatically.
The current code uses the FileConcern for this, and saves the reference to the file in the session data. However this has caused us some bugs in the past, and causes issues with edge cases like filling in the same form in two different browser tabs.
So instead I've introduced some code in the controller to implement this, by calling ActiveStorage::Blob.create_after_upload! with the file, and then associating the signed_id with the form using a separate attribute / hidden field (re-using the same attribute didn’t work as you then have 2 fields with the same name, which seemed to confuse Capybara at least). This is part of the controller for now, as it didn't feel it should be part of either the form class or the service object class - but we can refactor this into some shared behaviour.
The saving of the audit entry and the sending of the notification emails isn't yet part of the service class, as I didn't want to refactor too much at once, but I've left a TODO note, and hope to do this as part of a follow-up PR.
Screenshots
Before (2 pages)

After (1 page)

@StupidCodeFactory good catch! Didn't spot the validation as it was on the super class. Validation added to the form object in 1fa670f2ba06cc582b71787cbed786cbaf21eda5


Shader Object and VK_EXT_debug_utils

I tried to do the following in a VVL test
    const char *object_name = "bad_shader_object";
    VkDebugUtilsObjectNameInfoEXT name_info = vku::InitStructHelper();
    name_info.objectType = VK_OBJECT_TYPE_SHADER_EXT;
    name_info.objectHandle = (uint64_t)compShader.handle();
    name_info.pObjectName = object_name;
    vk::SetDebugUtilsObjectNameEXT(device(), &name_info);

and found it breaks with the VK_LAYER_KHRONOS_shader_object layer because it will send down a different handle to the driver (cc @ziga-lunarg)

So this crashes in DispatchSetDebugUtilsObjectNameEXT because it sends down VK_OBJECT_TYPE_SHADER_EXT to the driver, which in my case doesn't support it and crashes in the driver due to being unknown

diff --git a/layers/json/VkLayer_khronos_shader_object.json.in b/layers/json/VkLayer_khronos_shader_object.json.in
index cd3f58a..6a37477 100644
--- a/layers/json/VkLayer_khronos_shader_object.json.in
+++ b/layers/json/VkLayer_khronos_shader_object.json.in
@@ -70,6 +70,7 @@
                     "vkCmdSetViewportWithCountEXT",
                     "vkCreateShadersEXT",
                     "vkDestroyShaderEXT",
+                    "vkSetDebugUtilsObjectNameEXT",
                     "vkGetShaderBinaryDataEXT"
                 ]
             }
diff --git a/layers/shader_object/generated/shader_object_entry_points_x_macros.inl b/layers/shader_object/generated/shader_object_entry_points_x_macros.inl
index 45b713a..204f5eb 100644
--- a/layers/shader_object/generated/shader_object_entry_points_x_macros.inl
+++ b/layers/shader_object/generated/shader_object_entry_points_x_macros.inl
@@ -33,6 +33,7 @@
     ENTRY_POINT(DestroyDevice)\
     ENTRY_POINT(CreateShadersEXT)\
     ENTRY_POINT(DestroyShaderEXT)\
+    ENTRY_POINT(SetDebugUtilsObjectNameEXT)\
     ENTRY_POINT(CmdBindShadersEXT)\
     ENTRY_POINT(CmdBindPipeline)\
     ENTRY_POINT(CreateImageView)\
diff --git a/layers/shader_object/shader_object.cpp b/layers/shader_object/shader_object.cpp
index db5dfe0..923f2e4 100644
--- a/layers/shader_object/shader_object.cpp
+++ b/layers/shader_object/shader_object.cpp
+static VKAPI_ATTR VkResult VKAPI_CALL SetDebugUtilsObjectNameEXT(VkDevice device, const VkDebugUtilsObjectNameInfoEXT *pNameInfo) {
+    DeviceData& data = *device_data_map.Get(device);
+    VkResult result = VK_ERROR_OUT_OF_DEVICE_MEMORY; // valid error return
+    // The Validation Layers (if before this extension layer) will already have tracked this, but a driver might crash with it
+    if (pNameInfo && pNameInfo->objectType != VK_OBJECT_TYPE_SHADER_EXT) {
+        result = data.vtable.SetDebugUtilsObjectNameEXT(device, pNameInfo);
+    }
+    return result;
+}
+

This "fixes" the crash, the only issue is if you go VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_shader_object:VK_LAYER_KHRONOS_validation you won't see the name, so you just need to make sure your VVL layer is in front I guess (not sure if that is normal or not)


consulCatalog Provider: keep configuration / do not drop routes, middlewares or services on conflicting configurations

Traefik drops the configuration during rolling upgrades using consulCatalog

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you expect to see?
What I wish to see is an option on the consulCatalog Provider to enable "keeping routes, middlewares and services online if the case of conflicting configurations arises".
The issue is stated quite nicely here: https://community.traefik.io/t/router-defined-multiple-times-during-rolling-deployment/7517
The error in question here is:
Service defined multiple times with different configurations
This will happen if you do a rolling upgrade of services (we do it via HashiCorp Nomad) and the new version has a change in the routing rules, middlewares or loadbalancer definition and so on.
The Problem at hand is that in this case the complete router, middleware and/or service is dropped completely from traefik.
So, during the rolling upgrade - the service is not reachable at all but returns traefiks 404 (in the best case) or matches on another route.
Would it be possible to NOT drop everything as soon as this error/situation occurs but maybe wait (a configure-able amount of time) for the conflict to be resolved and keep the old/current configuration during that time ?
If this is possible, it would be great of course if this could be configured in the provider configuration.
Is anything like this planned ? Is it even possible ? And if not, could you please elaborate why ?

Hello @sirkjohannsen,
This behavior is indeed produced by design.
Traefik fetches all configurations from containers tags simultaneously and tries to merge them afterward.
When the merge operation detects identical resources definitions (routers, middlewares, services...), it drops them all.
So conflicts are avoided by dropping all resources defined multiple times.
As it has been pointed out, a solution is to name resources differently. Thus it implies having a process to handle that.
Another solution could be to manage a tag, that would act as a constraint to allow Traefik to consider the configuration from a container or not, by using the constraints option of the ConsulCatalog provider configuration.
In that scenario, you would "deactivate" and "activate" your containers during the rolling update.
Thus, the only zero downtime solution is to name routers, middlewares, and services differently, in order to avoid collisions.
Regarding a potential proposal, what do you suggest?
I understand, that keeping the previous configuration applied in case of conflicts until the convergence, would solve the rolling update case you've exposed.

Hello @rtribotte,
Thank you very much for your reply.
I already figured this behavior was on purpose.
Nonetheless - its not ideal for dynamic providers that support rolling updates ;)
We are using HashiCorp Nomad to orchestrate our services and love the fact the we provide the traefik routing information as tags alongside the job/service definition.
I do not think that any of the 2 proposed solutions will be possible for us.
Currently we are looking into using the consul-kv provider to define routers and middlewars that then route to the service @consulcatalog - sadly this will ALWAYS also generate the default router but it might work for us. Also - any configurations on service level will still have the same issue.
A proposed solution would be to add a parameter to the provider (e.g. keep_old_config_on_conflict ) or something similar that would implement the described behavior to keep the previous routes/middlewares (for a certain amount of time ?)  during a conflict situation.
Does that make sense ?
Many thanks again for your reply,
Sirk

@sirkjohannsen Thanks for your answer!
It's pretty clear, and it does make sense.
For a certain amount of time, I would say this would not be manageable. Also from Traefik point of view, it wouldn't be easy to consider a resource (e.g: a router) as the previous one and another that would be considered as the new one.
Also because potentially a router can be defined more than two times.
With those considerations, the solution I would come up with is to prevent the application of the configuration if a conflict is detected. This way, the previous configuration would remain applied until the new configuration converges to a state without conflicts.
The only grey area for which I'm wondering is what constraint prevents from being an acceptable solution to change the names of the routers and services before applying the rolling update?
I come back to this because this is obviously the simplest way to manage rolling updates.
Do you think that Traefik should be capable of an automatic naming that would allow avoiding conflicts in that case?

@rtribotte thinking about this more now - you might be right that defining new versions of routers and middlewares with a unique identifier might actually work. Still not sure about the service level configurations though but I will give it a try.
Btw, how will traefik behave if there are 2 routers with the same routing definition (e.g. Host=my-host) ?
Which router will take precedence ?

@sirkjohannsen Good question ;-)
In terms of routing, this can be an issue.
You'll have the priority of the router that will be taken into account to evaluate the first to try to match.
The priority is configurable, but it equals by default the length of the rule of the router.
A long rule is evaluated before a shorter rule.
Thus when the priority is the equivalent, the order is not guaranteed in any way, the first router in the list across routers with the same priority is simply evaluated first.
So if the configuration update doesn't alter the routing configuration or the priority, you'll end up with an uneven behavior if it creates two or more routers that can match the same request but route to different services.
In that case, controlling the priority is the only left option, but it's maybe not the more handful to manage.

@rtribotte I know about the priority but we highly rely on the automated calculation of that value running >60 services with different routing rules. So I would rather not start touching the priorities.
I was honestly afraid of having multiple routers with the the same routing rules because I already expected the behavior to be unclear.
Since in our case this only means we might route to the old or new version during deployment, it should not be an issue.
I will conduct some tests and evaluate if this solution will work for us.
Nevertheless - A builtin feature like described in this issue might make lives easier ;)

Many thanks @rtribotte
When using unique names for each service, router and middleware it works.
There are ofc multiple service, router and middleware definitions active during the deployment which seems a bit odd but it works nonetheless.
I will close this issue.

Hello, I'm sorry to bump a closed issue, but I've started running into this exact issue.
We have a service mesh that has a registered backend service for with Traefik, via the Consul Catalog provider. Our services also register themselves with our service mesh via an external API. All of our applications are deployed with Nomad and registered with Consul Catalog, but describe their public ingress using router blocks (on the service's job file) and point the router to the service mesh's backend service. If we change anything on our service tags for the service mesh, or for any of our services, that resource will be dropped from Traefik's configuration during deployments, when the new and old tags are registered in the catalog.
Due to our setup, we can't just rename the routers or services, as that would require redeploying our entire stack. I think the keep_old_config_on_conflict option is super reasonable and would like to try to PR that. Would you/the team be open to a PR to add that configuration knob?

@pirogoeth not sure if this helps you but we have / had the same issue. We still needed a "fixed" service name that could be discovered via consul service discovery - unrelated to traefik.
So we simply register 2 services per task now.
nomad job config snippet:
...
            # Service for traefik routing
            service {
                name = "my-service-<dynamic version>"
                port = "http"
                tags = [
                    "traefik.enable=true",
                    # traefik middlewares
                    "traefik.http.middlewares.my-service-<dynamic version>-ssl-headers.headers.sslredirect=true",
                    # traefik.routers
                    "traefik.http.routers.my-service-<dynamic version>.rule=Host(`my-host.xxx`)",
                    "traefik.http.routers.my-service-<dynamic version>.middlewares=my-service-<dynamic version>-ssl-headers",
                 ]
            }

            # service for service-discovery via consul
            service {
                name = "my-service"
                port = "http"
                tags = ["my-service"]
            }
...

In general I also still think it would be awesome to solve this on traefik side via a configuration option.

@sirkjohannsen thanks for the suggestion, I appreciate it. I'm not sure that fully solves our issue, but I'll certainly be chewing on this for a bit.


ioctl(BTRFS_IOC_TREE_SEARCH, subvol_id 359) ret=-1, error: Operation not permitted

After the other error was fixed, I now seem to get this error, same setup, I see the target directory created on the remote server and then this failure.
At subvol /home/user/lib/snapshot/20180418-002105
ioctl(BTRFS_IOC_TREE_SEARCH, subvol_id 359) ret=-1, error: Operation not permitted
ERROR: cannot resolve our subvol path
00:22:44  [ERROR]  Error during btrfs send / receive

Note: both servers were updated to 0.3.1 prior to testing.

Well, I can't really see what's causing the issue without having the exact options you invoked btrfs-backup with, but this one clearly is a permission problem. I guess that you don't log in as root at the remote server. Either do so, or configure sudo to not require a password for executing /bin/btrfs for that particular user and use the --ssh-sudo flag of btrfs-backup.
This is not a fault of btrfs-backup, it's rather a property of btrfs to not allow regular users to receive subvolumes, because the received stream could contain files that would normally not be readable/writable by that user due to permissions.

Maybe btrfs will get a mechanism to mitigate the problems with receiving snapshots as non-root users in the future, but afaik it's not possible at the moment.

This just as a reference, but for receiving the same reasons apply I think.
https://www.spinics.net/lists/linux-btrfs/msg63954.html

Ah it's probably the ssh-sudo flag, I don't use root but did enable passwordless sudo btrfs for that user. Will confirm this evening. Thank you for the help!

Genius @efficiosoft - somehow I completely missed that flag when reading up on the process, I now have snapshots working remotely.
Great work!

No problem and thanks for using btrfs-backup, good to hear that it's working now.
Best regards
Robert


im adding helloworld.c

added my helloworld

This PR is meant to be opened on https://github.com/cbitosc/CBIT-Hacktoberfest-Hackathon-Hello-World


Deprecated Warning in PHP 8.1

hash(): Passing null to parameter #2 ($data) of type string is deprecated in barzahlen/barzahlen-php/src/Middleware.php on line 27

I applied your suggestion

Thank you a lot for this contribution!


Cannot search using Cyrillic chars

Bug example - CodePen
The same issue has Lunr.js - https://github.com/olivernn/lunr.js/issues/318

I have no idea if it's still relevant to the asker, however, the info might be useful to other users as well.
I've encountered a similar problem with search using Hebrew chars. It's because the processing pipeline starts with a trimmer, which trims out every non word, in the current language, which by default is English. Therefore, any Hebrew/Cyrillic/any-non-Latin character is just removed.
The preferred solution is of course to add support to your desired language. However, a quick and dirty workaround is to add
index.pipeline.reset();

That empties the pipeline, effectively, treating the word as is, without any modifications. It's worse than having a proper full support for the language, but it's still good - it supports word expansion, booleans, sophisticated scoring, etc.


Enhance and expand tutorials

New tutorial "Getting started":

Explain how to use the different parts of the library to solve the popular Titanic example. The data is already available in the repository under docs/tutorials/data/titanic.csv.

Improve "Data Processing" tutorial:

How to use table transformers / invertible table transformers
How to transform a column with transform_column
Use Markdown headings so the table of contents is filled properly (see https://stdlib.safeds.com/en/stable/tutorials/data_visualization/)

Improve "Data Visualization" tutorial:

Add explanatory text to the sections "Boxplot" and beyond

Improve "Machine Learning" tutorial:

How to use metrics in general
Which metrics are available
What do the different metrics mean (formulas) and when should they be used
Also useful to explain things like confusion matrix, false/true positive/negatives. Refer to the glossary by @patrikguempel for more information.

Added a new tutorial getting_started.ipynb to docs/tutorials to get users to use safeds to reimplement the famous example of the titanic. In the tutorial the user will use the features age and travel_class to predict the survival of passengers. Throughout the tutorial, users will get familiar with:
-Creating a table from a csv File.
-Using OneHotEncoder to transform columns.
-Creating a tagged table out of a table.
-Using RandomForest as a classification model.
-Predicting values using predict function.
Modified the tutorial data_processing.ipynb by adding examples to apply the following:

Transforming table using Imputer using Constant(0) as strategy.
Transforming table using LabelEncoder.
Transforming table using OneHotEncoder.
Transforming table using RangeScaler, using range 0.0-1.0.
Transforming table using StandardScaler.
Transforming using the function transform_column of the Tableclass.

Modified the data_visualization.ipynb tutorial in docs/tutorials with detailed explanations for users to achieve a more comprehensive view of the box plots, histograms, and scatter plot. The user will learn how to interpret table data with the help of  different column plots such as age, survived, parents/children.
So, the following sections of the tutorial were modified to achieve this:

High level overview of a Box plot.
Box plot explanations of all numerical columns.
Definition and basic information of a Histogram.
Histogram explanations of all columns.
Understanding a Scatter plot with the help of the age and fare columns.

Added a new section Metrics in machine_learning.ipynb tutorial. It detailly explains ML-metrics such as Confusion Matrix, Accuracy,  F1-score, Precision, and Recall. The user will understand how to better interpret a confusion matrix  and when to use the different metrics used to evaluate a machine learning model.
The following sub-sections were added:

High level overview of Machine Learning Metrics
The different components of every metric
Confusion Matrix: Definition, example code and code Interpretation.
Accuracy: Definition, example code and code Interpretation.
F1-score: Definition, example code and code Interpretation.
Precision: Definition, example code and code Interpretation.
Recall: Definition, example code and code Interpretation.
When to use each of these metrics.

:tada: This issue has been resolved in version 0.15.0 :tada:
The release is available on:

v0.15.0
GitHub release

Your semantic-release bot :package::rocket:


How to calculate spline length?

Hello everyone, im struggling while trying to find the length of a spline that will be inputted by the user. How can I achieve it?

i'll check it out, thanks!


DataPicker (more dynamic DatePicker)

#4549 Summary

What is the change?
This is a new component. The reason is we can reuse code that we already have in the date picker. The DataPicker will have the similar look and functions as DatePicker, but the creator can have the chance to replace date/month with their own data. One good example is the Emoji/Icon Picker.
Why should it go in Clarity?
I believe this is one essential component and enterprise website usually have toolbars, which needs an icon picker most of the time if it's a web-app.
Does this change impact existing behaviors? If so how?
It doesn't chance anything.
If this change introduces a new behavior, is this behavior accessible?
No new behavior from DatePicker.

Use case
So developers can have their own data picker without using any external library.
Examples
I don't know much about designing but I can try if you want. Otherwise, imagine no date, just emojis (small data) and months are categories.
API
API is the same as DatePicker
Implementation plan
I will be examining the DatePicker code and build from there.
Conclusion
It would be helpful if I have some help from anyone in the Clarity Team. This is such an amazing library so thank you all and I want to contribute to it too. Please let me know if you have any suggestions!

Hi @quanglddev, thanks for making a suggestion. I am not very clear on the details here or the use case. Why is a generic 'picker' something that Clarity should provide? The date picker is built specifically for picking dates, so I don't think its likely to work by trying to build something similar to it.
To move forward, I need a lot more detail about why this is valuable and the use cases before we could consider this as something that Clarity should adopt. Then we'd have to review the design and use cases carefully before we develop anything.

Hi @gnomeontherun !
Thank you so much for the reply. I believe this is extremely helpful since there are many applications to it: emoji picker, color pickers etc.
You can find a similar version here: https://github.com/tech-advantage/ngx-icon-picker
Use cases:

The component gets a list of string and display it in a grid and have pages to navigate around.
Optional categories also so user can switch around categories.

I’m not experienced with use cases so please follow up if you want more information. Thank you.

The feature request here has been captured into our list and we’re going to take it into consideration as we develop Clarity Core capabilities. In an effort to clean up our backlog and focus our attention, I’m going to close this as captured in our feature requests. Please follow our development and releases to see when we release relevant components to make this possible. Future feature requests can be made in our GitHub Discussions.


Reduce allocations in camera_system

Objective

camera_system allocates at least 4 times per invocation.

Solution

Cache buffers for reuse.

Todo

[ ] Benchmarks

I am clearing at the start of the system.

You absolutely are, I completely missed that!


updated single-box-shadow() for compass 1.0.1

The $color argument for single-box-shadow is now the 5th argument instead of the 1st.

Will be fixed with the above commit


Write a C++ program to find the average of all numbers in an array

Description
Write a C++ program to find the average of all numbers in an array
Input  : [1, 2, 3, 4, 5]
Output : 3

How to contribute

Save the solution in program/FindTheAverageOfAllNumbersInAnArray.cpp file
Add FindTheAverageOfAllNumbersInAnArray.cpp file in find-the-average-of-all-numbers-in-an-array folder

Hi @harshraj8843, this issue is already solved - reference PR #2004

Hey @hi-Kartik2004
PR #2004 is linked to issue #1858
This issue is similar to #1858 but for different purpose

Hey can I solve this issue

Hey can I solve this issue
Hi @AbhinavPrabhat01, thanks for showing your interest in contributing, you can type !assign on any good first issues which are not already assigned to someone else, in this case, this issue is open and not assigned to anyone, and if you wish to contribute just type !assign.


MAC App was crashed in full screen size (inAppWebBrowser MAC)

Platform : MAC
Bug Description :
I used inAppWebBrowser for MAC.
I try to close the inAppWebBrowser in full screen, app was crashed on MAC.
But it's work on minimum screen size.
Logs
(macOS) MyInAppBrowser ID---------- calling "onExit" using {}
ERROR: Expected a non-nil view or subclass override
ReloadButton

Please give the solution for this issue quickly.