[Feature] Implementation of multi-gpu KV cache (RingAttention)

I'll work through adding it to quantized llama first, as I know that architecture the most. Link to the paper: https://arxiv.org/abs/2310.01889

I've been researching the algorithm further, and I'm thinking I'm going to have a problem implementing this with Rust. To start, based on my understanding, I'd need to split the input into different Tensors, with each split tensor on a different GPU. Pytorch has select https://pytorch.org/docs/stable/generated/torch.select.html, but doesn't look like candle has an equivalent. Is that correct?

To split the input into different tensors, I would use the narrow method to split along different ranges. If you want something a bit more flexible, IndexOp is probably what you want.
To implement torch.select, I'd use IndexOp (.i) to extract at the given dimension.
Are there any other problems? Happy to help!

Thanks for the help! That's it at the moment.
I'm just trying to implement the first part of the algorithm, which is the sequence parallelism. As I work through it, I'll definitely have more questions; it's just going to be slow as I'm really digging deep into this for the first time.

I've been trying to implement this algorithm from the paper https://arxiv.org/pdf/2310.01889, and it really isn't working.

The KV cache isn't being split so that's a big problem, but I'm more concerned with the garbage output that I'm getting. If I do a PR, can you give it a look and let me know what looks incorrect?

Just adding a little more info.
I think the biggest problem I'm facing is that the KV cache needs to cycle between GPUs. I'm trying to do this with a for loop to get something working, but I don't think it is correct.
The device that the KV cache is using is set based on the Key and Value tensor devices. It's almost like I need a method to move the Tensors to new Devices. Here is my implementation so far:
let num_caches = self.kv_caches.len();
let mut accumulated_attention: Option<Tensor> = None;

for cache_rotation in 0..num_caches {
    let cache_idx = (chunk_idx + cache_rotation) % num_caches;
    let kv_cache = &self.kv_caches[cache_idx];
    let mut cache = kv_cache.lock();

    let device_chunk = chunk.device();

    // Determine the device of the cache
    let cache_device = cache.iter().find_map(|opt| {
        opt.as_ref().map(|(k, _)| k.device().clone())
    }).unwrap_or_else(|| device_chunk.clone());

    let mask = CausalMasker.make_causal_mask_as_attn_bias(
        input_ids,
        metadata
            .as_ref()
            .map(|(_, _)| &seqlen_offsets as &dyn PastKvLenCache)
            .unwrap_or(&*cache as &dyn PastKvLenCache),
        chunk.dtype(),
        self.blocks[0].attn.num_attention_heads,
    )?;

    let mut x = self.mapper.map(chunk.to_device(&cache_device)?, block_idx)?;

    x = block.forward(
        &x,
        &mask.clone().map(|m| m.to_device(x.device()).unwrap()),
        seqlen_offsets,
        start_offsets_kernel.clone(),
        block_idx,
        &mut cache,
        metadata
            .as_mut()
            .map(|(kv_cache, metadata)| (kv_cache[block_idx].clone(), &mut **metadata)),
    )?;

    // Accumulate attention results
    if let Some(ref mut acc) = accumulated_attention {
        *acc = acc.add(&x.to_device(acc.device())?)?;
    } else {
        accumulated_attention = Some(x);
    }

Let me know if you'd like to see the full code so far in a PR. I've made most of the changes in mistralrs-core/src/models/llama.rs.

@joshpopelka20 yes if you do a PR, I can absolutely take a look.

It's almost like I need a method to move the Tensors to new Devices.

Could you use Tensor::to_device? For the KV cache to cycle between 2 devices, I would check out the Cache struct. This struct has no idea of device mapping though, so it can innately handle KV cache split on multiple GPUs.

@EricLBuehler thanks for the reply.
For:

Could you use Tensor::to_device?

I'm already using this for the chunks. From this code:
let mut x = self.mapper.map(chunk.to_device(&cache_device)?, block_idx)?;, you can see I'm setting the chunk to the kv_cache device (the chunk is just the seq_len split into num_devices, in my case, four.
            for j in 0..num_devices {
                let start = j * chunk_size;
                let end = if j == num_devices - 1 {
                    seq_len
                } else {
                    (j+ 1) * chunk_size
                };

                let chunk = x.i((.., start..end,..))?; // use IndexOps
                let device = &self.cuda_devices[j];
                chunks.push(chunk.to_device(&device)?);
            }

I would check out the Cache struct.

I asked about a method to move the KV cache because that's what the algorithm does. I'm trying to do it by moving the chunks device, but I'd think if I could have a way to move the KV cache device, I'd get better results. The only method that looks close is:
fn clone_in_cache(
    num_hidden_layers: usize,
    cache: &mut LayerCaches,
    seqs: &mut [&mut crate::sequence::Sequence],
    src: SeqCache,
)

I think I need a method that takes in two kv caches and the new device, then clones the first kv cache data and moves it to the second kv cache on the new device. So basically, is there a method to clone the key and value tensors? I can use to_device for changing devices.

I think I have enough info now. Currently, I'm working through doing some instruction fine-tuning, which is taking all my time. I should be able to work on this again starting next week.

@joshpopelka20 sounds good - please feel free to open a PR so I can take a look!

I think I need a method that takes in two kv caches and the new device, then clones the first kv cache data and moves it to the second kv cache on the new device. So basically, is there a method to clone the key and value tensors? I can use to_device for changing devices.

Modifications like this would be done in the Cache struct. You could add a method to it which tells it to map the devices for its KV cache, would that work?

Modifications like this would be done in the Cache struct. You could add a method to it which tells it to map the devices for its KV cache

That's the plan. After I add that, I'll create the PR.

 My problem is with the Tensor operations. I have to clone 'x' to insert into the chunks Vec and clone 'chunk' in the map function. When I do this, the output isn't coherent. For some reason, that operation isn't copying the Tensor properly. See this code sample:
let mut chunks: Vec<Tensor> = Vec::with_capacity(num_devices);
chunks.push(x.clone());

let mut cache = self.kv_caches[0].lock();
let mask = CausalMasker.make_causal_mask_as_attn_bias(
    input_ids,
    metadata
        .as_ref()
        .map(|(_, _)| &seqlen_offsets as &dyn PastKvLenCache)
        .unwrap_or(&*cache as &dyn PastKvLenCache),
    chunks[0].dtype(),
    self.blocks[0].attn.num_attention_heads,
)?;
for (block_idx, block) in self.blocks.iter().enumerate() {
    x = self.mapper.map(chunks[0].clone(), block_idx)?;

When I remove the clone function, I get these errors:

This is the image of the garbled output:

Any help would be appreciated.

The issue seems to be how the clone assigns to the device. Here, you can see that the 'x' tensor is split across devices, but the 'chunk' is on the same device:

I don't think I can use that method as is. It needs something to clone to the correct device. Here is the code snippet for reference:
for (block_idx, block) in self.blocks.iter().enumerate() {
    // x = self.mapper.map(x, block_idx)?;
    // x = self.mapper.map(&chunks[0], block_idx)?;
    println!("x device {:?}", x.device());
    println!("chunk device {:?}", chunks[0].device());
    x = self.mapper.map(chunks[0].clone(), block_idx)?;

Can you please open a PR so I can take a look at this? It's OK if it's unfinished for now, I can just take a look at what is going on right now. I'll be able to help better that way!

I've add PR #684.
I restarted everything to try to see where the errors are, so it's very minimal. The biggest issue is the garbled output at the moment. After I figure that out, I can start adding the IndexOp and multiple KV caches.

Ok great! I'll take a look.

I've added a commit with the Sequence Parallelism code. Also, documented a few things in the comments of the PR. Please review when you have the time.

I'm stuck with trying to implement this, and need some help. My current issue is that the blocks and chunks are on different devices. Here is the error:

I'm not sure how to proceed; as per the algorithm, the input sequence should be split across devices. But, candle won't allow operations across devices. Any help would be appreciated.

Researching further, I'm thinking Ring Attention is incompatible with the pipeline parallelism implementation since the layers are split across devices; most likely need to have the model shared across devices so I can run the operations that are failing. Maybe, I'll need to wait for tensor parallelism #617 .

@joshpopelka20 that's probably true. The timeline for that is a bit undefined, it needs some time to have a clean implementation here, but I want to get #617 and #684 implemented for sure.
I've been focusing on things around quantization recently, tensor parallelism is going to build off of some of that infra.


run_exports

building against zeromq will have a runtime dependency of zeromq>=4.2.5,<4.3

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


How to reduce data disk usage in Azure

Question
Azure VMs have a limit on data disks (from 2 to 32) depending on their size. If using RexRay with Docker Swarm mode and thus deploying a lot of containers, it's very easy to exceed the data disk limit of the machines, since every Docker volume is attached to a data disk on the container storage account. Considering that every volume is also attached to every node of the swarm, scaling horizontally VMs doesn't help much and the only (expensive) workaround is using a bigger size VM.
Consider that a D14 VM costs around 1000$ per month and has maximum 32 data disk.
Is there an alternative way to configure Rexray / libStorage in order to be more scalable in this sense?

@cecchisandrone I know this an old issue, and apologies for not responding before...
Right now, no, but we've talked about adding the ability to "carve up" a volume into subdirectories and mount those as volumes instead. I don't have a timeline to give you yet, though.


Refactor project to use typescript

Addresses #20

Migrate the project to Typescript
Refactor connection to the database
Remove in-memory struct used for managing rides
Nits

@saviomuniz, if you want to, feel free to review it. I'd appreciate your opinion on it, since it's been a long time since I worked with Typescript


VoxelGI: Incorrect reflections after a rebake

Tested versions
v4.3.dev.custom_build [4a0160241]
System information
Godot v4.3.dev (4a0160241) - macOS 14.4.0 - Vulkan (Forward+) - integrated Apple M1 Max - Apple M1 Max (10 Threads)
Issue description
Voxel reflections don't look correct after a rebake. They get fixed after the editor restart though
https://github.com/godotengine/godot/assets/60579014/b4f8cebb-1a55-4f03-9039-bab05d466e6d
Steps to reproduce
Open the MRP, move the cube and rebake
Minimal reproduction project (MRP)
MRP_voxelgi.zip

I can confirm this on 4.2.1.stable and 4.3.dev b091162a9 (Linux, GeForce RTX 4090 with NVIDIA 550.67).
MRP that can be opened on Godot versions older than 4.3: MRP_voxelgi_pre_4.3.zip


[question] Difference between using a mounted NFS directory vs netshare plugin

I have been using local volumes before, in a docker environment without using swarm. Now I started a new project with a swarm cluster and I am exploring my options. A shared NFS share seems like a quite simple setup, that will work as long as I do not care having the storage as my SPOF.
Since I used a NFS mounted directory before without using any plugin and it worked well, I am wondering, what are the benefits of using the netshare plugin? what can I do with it that its not possible or inconvenient to do without it just creating a local docker volume using the path to the mounted NFS share?

For a single application host-level NFS volumes are probably ok.   But once you start running many different applications/services and you want them to have their own "storage" it becomes a management nightmare.  This the plugin links the volume within the container thats requesting it.


requirements file

@vinisalazar:

The requirements file should explicitly list the requirements. Having it as is (only content is -e ., presumably for
pip install or a setup command) is bad practice. Either remove this file or modify it to list the requirements.

Previously I had the requirements listed in the file, until I found somewhere written (which I can't find right now), that the current practice was exactly that one, in order to avoid duplications.
Where can I find an up-to-date description of the current good practice regarding requirements listing?

38d06a0 fixes this, by listing the requirements in the files


Save should accept string or array in order to do PATCH instead of PUT

In #584 you decided against PATCH support within .save(), because you said you would need to keep a copy of the complete object in $rootScope. But sometimes you know beforehand that some elements are just a partial element. Say you have a REST API like /api/users/123?fields=name,age than you get some object like { "name": "Pipo", "age": 25 } and you know that the complete element would have even more properties. It would be great to use PATCH instead of PUT within .save() then.

I like idea of passing array of properties or a string of single property to be passed which will execute PATCH request instead of PUT. Will throw an error if ID is empty.
``js
user.save('name');
/ PATCH /users {name: ...}
user.save(['name', 'id']);
/ PATCH /users {name: ..., id: ...}

if that's the case, IMO it should use either array or string notation, not both - just saying :-)

Yeah, maybe, then string can be left for other fancy parameters we might add at some point :D

well, i see nothing wrong with
model.save('name');
model.save('name', 'age');

Yeah, that's true. Let's get back to this discussion once V2 is on its way.


[edpm_deploy]Allow passing extra nova-compute config

The install_yamls edpm_deploy_prep target can take an config file path via DATAPLANE_EXTRA_NOVA_CONFIG_FILE ENV variable. Expose this capability via the new cifmw_edpm_deploy_nova_compute_extra_config ansible variable. This way the zuul job definitions can define the following to pass a config snippet to nova-compute:
      vars:
        cifmw_edpm_deploy_nova_compute_extra_config: |
          [libvirt]
          cpu_mode = custom
          cpu_models = Nehalem

Depends-On: https://github.com/openstack-k8s-operators/install_yamls/pull/658 (merged)
As a pull request owner and reviewers, I checked that:

[x] Appropriate testing is done and actually running
[x] Appropriate documentation exists and/or is up-to-date:

[x] README in the role
[x] Content of the docs/source is reflecting the changes

the install_yamls dependency landed. And the CI in this PR also shows now the extra (still empty) config file https://logserver.rdoproject.org/41/941/011e1986024deae049ef8f12de02ec18effd273f/github-check/podified-multinode-edpm-deployment-crc/b100990/controller/ci-framework-data/logs/<IP_ADDRESS>/openstack/config/nova/

And we have proof that it works end-to-end: https://github.com/openstack-k8s-operators/nova-operator/pull/621#discussion_r1428412743

/approve


Optimize display of Day-of-the-Week filter

@johnreynolds123, what were you envisioning here?
-Tackle


feat(nameCache) Retain the nameCache option as an object reference

I have a use case where I need to pipe to gulp-minify with different options for different files, but I need a common nameCache.
This could be a breaking change, but I think it more accurately mirrors the interface provided by uglify itself.

Coverage increased (+0.5%) to 96.296% when pulling d66762f327b2d0918521801a77d7e9147fc0a648 on krisdages:feature/retain-name-cache into 14f1d7190fc94efd6f29b2e16957418bbf1a3adf on terinjokes:master.

I've not seen this option before, can you show me its use?
On Oct 30, 2017 4:26 PM, "Coveralls"<EMAIL_ADDRESS>wrote:

[image: Coverage Status] https://coveralls.io/builds/13961277
Coverage increased (+0.5%) to 96.296% when pulling d66762f
https://github.com/terinjokes/gulp-uglify/commit/d66762f327b2d0918521801a77d7e9147fc0a648
on krisdages:feature/retain-name-cache into 14f1d71
https://github.com/terinjokes/gulp-uglify/commit/14f1d7190fc94efd6f29b2e16957418bbf1a3adf
on terinjokes:master.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/terinjokes/gulp-uglify/pull/318#issuecomment-340614809,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAQsZbhZz4AaZEwj-xn3Zrbl3UgmnK_vks5sxls3gaJpZM4QL_dF
.

Actually there may be some issues with this, ended up with some overlapping mangled values.  I think this PR may be premature, When I figure out what's going on I'll update.
Basically uglify allows you to pass in an object that it populates with the values that it generates for mangled properties and identifiers.  You can then pass that object in to uglify again for additional files to make sure that it reuses the same mangled value for the same name.  It's supposed to be the same as passing the --name-cache=file on the command line.

It looks like the bug I was seeing is in uglify itself, which potentially creates duplicate tokens in the situation I am using it for.
The behavior of the nameCache option is correct here.... Pass { nameCache: { } } as an option to gulp-uglify as it currently stands and it will remain empty after minification is complete.  Try it with the change in this PR and the object will be populated as described in the uglify documentation:
nameCache (default null) - pass an empty object {} or a previously used nameCache object if you wish to cache mangled variable and property names across multiple invocations of minify(). Note: this is a read/write property. minify() will read the name cache state of this object and update it during minification so that it may be reused or externally persisted by the user.
Thanks!

It would be great if you could give provide a little more prose in your commit message explaining this change. The standard format here is to use "minifier" as the module and use only lowercase (unless describing a proper noun).

+1.
In my TypeScript project, I need to uglify 3 streams (3 groups of files), and share a nameCache object among them, just like
https://github.com/gdh1995/vimium-c/blob/f60db7a2798e110fa867c7d13d028bb5b2c4ae05/Gulpfile.js#L61-L75
But currently gulp-uglify deeply copies my options in lib/minify.js, so my temporary solution is to patch the file before require('gulp-uglify'):
https://github.com/gdh1995/vimium-c/blob/f60db7a2798e110fa867c7d13d028bb5b2c4ae05/Gulpfile.js#L683-L698
If this PR gets merged, it will help a lot.


Document extism.Json

Does this look correct @zshipko ? Is there maybe a test i can grab?

The macos install script test has been flaky for some reason, I need to spend some time fixing CI for this repo


ref(core): clean setup of core

The PR includes:

build steps
testing setup
eslint setup and config
changed spaces to tabs
events that other projects use
tests for these files

We won't do this this way.


Don't use AtomicBool in seg_queue::Segment

The ready field of seg_queue::Segment is horribly wasteful:
    ready: [AtomicBool; SEG_SIZE],

AtomicBool uses usize internally, so on 64-bit platforms and SEG_SIZE = 32, that's 256 bytes, 2048 bits, for only 32 bits' worth of information. In each segment.
I think we can do much better here, using AtomicUsize as the storage for an atomically accessed bit-set. Since we only need 32 bits of information (true or false for each cell), we can get away with a single AtomicUsize, a whopping 248 bytes' reduction in size per segment for little additional cost elsewhere. For very long queues, this could mean a significant savings in memory usage, which also means more cache-friendliness.
There might very well be some nuances I'm overlooking, but it seems as though the semantics should remain almost exactly the same. The one thing I can see is increased contention across threads, but I don't know if that'd be an issue or not. Clearly some benchmarks are needed here.

I did some basic benchmarks, and using a single AtomicUsize is about twice as slow as an array of AtomicBool. I expected it to be faster due to better cache locality.

Do you know why? (Or could you post the benchmarks?)
I would guess that you are hitting false sharing. Basically for atomic
variables, cache lines must have a consistent state between the caches of
each CPU core.
That is, writes to the cache line containing a single AtomicUsize must be
synchronized between core caches. If two writes to it happen in different
caches at the same "synchronization step", the operation is retried until
this succeeds. This can make microbenchmarks much slower. Furthermore, any
other write to the cache line where the AtomicUsize lives might be
responsible for this, so it would be useful to know if there is anything on
that cache line that is also hot. Adding some padding (a cache line width)
before an after the AtomicUsize should be enough to study this.
If you are really hitting false sharing, the AtomicBool array might be
hitting it too. It would be interesting to study if having a single
AtomicBool per cache line can further improve the performance at the cost
of ~4-8x more memory.
TL;DR: unless you are modifying a group of atomic variables from the same
CPU core (i.e. your threads are pin), you generally want your atomic
variables to be allocated in different cache lines if they are under a lot
of contention from multiple threads (which is typically the case in
microbenchmarks, and might not be representative of real applications).
On Fri, Nov 13, 2015 at 12:16 AM, Austin Bonander<EMAIL_ADDRESS>wrote:

I did some basic benchmarks, and using a single AtomicUsize is about
twice as slow as an array of AtomicBool. I expected it to be faster due
to better cache locality.
—
Reply to this email directly or view it on GitHub
https://github.com/aturon/crossbeam/issues/19#issuecomment-156267706.

@gnzlbg Here is the gist, this is the lib.rs of a Cargo project with lazy_static = "*" in its dev dependencies:
https://gist.github.com/cybergeek94/6ea304b19781b670a26e
I just now added the padding to AtomicBitSet, but it didn't make a difference at all. 8 usizes should be 64 bytes, the length of a x86 cache line, on my machine, so that's taking a whole cache line for itself.
These are the bench results:
test test::bench_atomic_bit_set    ... bench:       1,186 ns/iter (+/- 171)
test test::bench_atomic_bool_array ... bench:         485 ns/iter (+/- 8)

It's curious. I'm not an assembly guru but I don't think I see any atomic operations actually taking place in bench_atomic_bool_array. bench_atomic_bit_set contains lock instructions which, according to a Google search. lock the cache line like you said. The array doesn't do that.

It just occurred to me that I'm not even testing concurrent access. Maybe
I'll add some duplicate benches for some contention.
On Nov 13, 2015 3:49 AM, "gnzlbg"<EMAIL_ADDRESS>wrote:

Do you know why? (Or could you post the benchmarks?)
I would guess that you are hitting false sharing. Basically for atomic
variables, cache lines must have a consistent state between the caches of
each CPU core.
That is, writes to the cache line containing a single AtomicUsize must be
synchronized between core caches. If two writes to it happen in different
caches at the same "synchronization step", the operation is retried until
this succeeds. This can make microbenchmarks much slower. Furthermore, any
other write to the cache line where the AtomicUsize lives might be
responsible for this, so it would be useful to know if there is anything on
that cache line that is also hot. Adding some padding (a cache line width)
before an after the AtomicUsize should be enough to study this.
If you are really hitting false sharing, the AtomicBool array might be
hitting it too. It would be interesting to study if having a single
AtomicBool per cache line can further improve the performance at the cost
of ~4-8x more memory.
TL;DR: unless you are modifying a group of atomic variables from the same
CPU core (i.e. your threads are pin), you generally want your atomic
variables to be allocated in different cache lines if they are under a lot
of contention from multiple threads (which is typically the case in
microbenchmarks, and might not be representative of real applications).
On Fri, Nov 13, 2015 at 12:16 AM, Austin Bonander <
<EMAIL_ADDRESS>wrote:

I did some basic benchmarks, and using a single AtomicUsize is about
twice as slow as an array of AtomicBool. I expected it to be faster due
to better cache locality.
—
Reply to this email directly or view it on GitHub
https://github.com/aturon/crossbeam/issues/19#issuecomment-156267706.

—
Reply to this email directly or view it on GitHub
https://github.com/aturon/crossbeam/issues/19#issuecomment-156408343.

@gnzlbg Here is the gist, this is the lib.rs of a Cargo project with lazy_static = "*" in its dev dependencies:
https://gist.github.com/cybergeek94/6ea304b19781b670a26e

But that's all serial right? Then the problem cannot be about false sharing.
Can you post the assembly? The bit set solution cannot be vectorized, but the array solution can and probably is being vectorized.

I just now added the padding to AtomicBitSet, but it didn't make a difference at all. 8 usizes should be 64 bytes, the length of a x86 cache line, on my machine, so that's taking a whole cache line for itself.

Yes, in x86 a cache line is 8 * 64bits. The size of AtomicBitSet without padding is only 64bit, so in general it will share a cache line with other stuff. The test of making sure that AtomicBitSet is "alone" in a single cache line is only to discard that the slow down comes from anything else it might be sharing the cache line with.
I forgot to mention before that padding is not enough. The AtomicBitSet can still be allocated between two cache lines. So besides padding, one needs to require AtomicBitSet to be aligned to a 64byte boundary (its memory address must be a multiple of the cache line size).
If this makes no difference at all, then the only thing we know is that iff this is due to false sharing, the false sharing is occurring only within the AtomicBitSet and nothing else is involved in it.

Just looking at the bitset implementation, there's going to be really bad CAS contention on the atomic or -  Even a successful CAS can be reasonably expensive, and failed ones are outright bad.
You can run this for some benchmarks on your machine of how various x86 atomics fare under different levels on contention - long story short, its not good with no contention and really bad with contention.
IMHO, each element in the array should be a struct containing the ready variable and the element, that will get some better cache separation and reads on one will usually read the other as well.

Yeah, I realize that my intentions were misguided. Forget I said anything.

Actually, I think they are right - that array is really wasteful and there's likely a better way to do this.


Changes scripts, added files

I added the files that were downloaded from other repositories, to this one. Changed the scripts, broke them into two-part install for each component, it seems to installing properly now, even nDPI.
Fixed pfbridge automatically working, now it should start at boot.

@skolome There seems to be an error in the build process:
Sending build context to Docker daemon 271.4 MB

16:06:43 Step 1 : LS
16:06:43 Unknown instruction: LS
16:06:43 Build step 'Execute shell' marked build as failure

You can check the automated build process by executing build.shin the vnfs/ folder.
Can you als fetch the latest updates from the repo to resolve the merge conflicts?

updated branch, fixed Dockerfile

Thanks. Lets wait until Jenkins is back online on Monday :-)

test this please

test this please


Update .screenrc, package.json and correct readme typos

@glennsl , thanks for merging my last pull request.  I noticed a couple of typos in the Readme file and updated .screenrc to reflect dependencies on rescript.  All tests pass.  Please consider merging this into the current release.

BTW, would you mind updating the about section of the repo description to
Rescript or BuckleScript bindings for Jest
*Topics*
javascript  jest  bucklescript  reasonml rescript

Done. Thanks!


EB-396: Generate assembly aliases as part of the data build

Context
Alias files for ENA assemblies are currently being generated by manually invoking get_aliases_from_ENA_fasta.py. This presents two disadvantages:

It adds a manual step to our workflow
The aliases are made available to JBrowse by hosting them at a conveniently accessible location. Since neither Google Drive nor Nextcloud proved suitable, we fallback to scripts/data_stewardship_alias_files_temp_storage in this repository. This forces us to check in sometimes large files to version control unnecessarily.

Contribution
This PR introduces a make aliases recipe that will output one aliases.txt per species. Thus, make build will store the alias file alongside other genomic assets to be deployed on k8s.
Implementation

The logic in get_aliases_from_ENA_fasta.py was ported to a sed script in scripts/aliases. This is a brittle move perhaps, so I am open to keeping the logic in Python. This solution got me started quickly however, since the Docker image we use for the build does not currently have Python installed. And I learned how to branch in sed which was kinda 😎

make aliases generates one aliases.txt per species. If the species has multiple assemblies,  all the aliases will be in this file. It is slightly inefficient since aliases.txt will be empty for non-ENA assemblies, but I thought the gained simplicity was worth it.

If an assembly's alias field is omitted in config.yml, it defaults to aliases.txt (see 0bc26e)

Testing

Unit test are found in tests/bats/aliases but running them requires some setup that I can help you with.
Otherwise, for example, at the root of the repository:

# Optionally, you can add our scripts directory to your executable search path
PATH=$(pwd)/scripts:${PATH}
# After setting SPECIES in the environment, all future make commands will operate on it
export SPECIES=linume_tenue
dockermake aliases
dockermake jbrowe-config

I think if we see down the line a need for quite a bit more scripting it could be worth considering adding Python to the image to make it more flexible.

I completely agree, this is a tipping point. Should we need, for example, to support aliasing another flavor of FASTA headers, then I would switch to Python.

I think having an empty alias.txt file for species without an ENA file is not the end of the world

Agreed. The more annoying inefficiency is that the mere streaming of a large FASTA through sed, even when no substitutions are performed, takes quite a lot of time. Again, not the end of the world.

I got the following error (it repeats many times):
Indexing data/meganyctiphanes_norvegica/annotations.protein_coding_gene_models.non_redundant_added_stop_codons.gff.bgz
[E::get_intv] Failed to parse TBX_GENERIC, was wrong -p [type] used?
The offending line was: ""

Thanks for surfacing this. Those are in fact warnings emitted by tabix that cannot quite stomach the empty lines found in the Krill GFF, but produces a valid index nonetheless . This originally motivated EB-378 and I could update the issue description thanks to your detailed log trace.

Good to hear you're both aware of and on top of this issue. No need to try and "fix" it in this PR then.
@brinkdp

Sounds good to me. For the aliases, this implementation seem to cover a lot of ground.
But for the upcoming slew of helper scripts that I have started working on with Angela, I think they will be mainly in python. I am very keen to learn more about docker, and it would be a fun exercise for me to set up a docker file for the dependencies of the data stewardship scripts. Are there any reasons for not including python in the docker images, apart for it not being used in the data builder processes that we are talking about here?

Nothing wrong with having a python docker image for the helper scripts and as you say this could/should be a separate docker image and not shared with the data builder image. By all means create a new dockerfile if you would like to :). P.S. If you plan to have a requirements.txt file for all the dependencies of these scripts in this case you could name it something like requirements-[distinguishing-name].txt (e.g. see how they do it here ).

can we assume that contig:  would either never or always be present in the headers ?

Indeed, this can not be assumed. From my anecdotal experience, contig: is present in most of the assemblies that we have encountered, but not all. I could hypothesize that the assemblies that have been produced and brokered with help from SciLifeLab staff all the substring, but I do not know for certain.

To manually handle alias patches, I have a few ideas that we could discuss. One of them could be an approach akin to how config.json is handled: we can put a default aliases.txt in config that would be extended by whatever can be parsed from FASTA headers. In fact, going that way, we could move the Clouded Apollo aliases to config/parnassius_mnemosyne/aliases.txt. Do you see what I mean ?

I like this idea! Adopting this philosophy also to the aliases would be good for consistency in the work-flow.

I like this idea! Adopting this philosophy also to the aliases would be good for consistency in the work-flow.

I have started an implementation, but will keep it for a separate PR. Stay tuned :)
I'm merging this now after having cleaned up the history a wee bit  🚀


terminal takes input right after the arrow instead of leaving a space

it sounds unimportant but it genuinely makes the terminal looks so bad, i tried to edit the __flseg_arrow.fish file so it leaves a space before taking input but i couldn't manage to do it

You can try adding a 'space' segment after your arrow. It should do what you want. It closes the previous segment and just adds a space. The effect you are seeing here also depends on how your terminal renders the font, on my system if usually still leaves enough of a gap between the arrow and the first input so yeah. Hope this helps.


Added new notebook with DIA Run2 introduction

Adding notebook for mini sprint session with DIA

hi @BrunoSanchez and @wmwv, are the things in place to make these changes/updates given the recent progress?

I am not sure. I think I have resolved everything requested.
Michael still needs to approve on his side I think.

@BrunoSanchez I think we can merge this. Can you first clear the output of the notebook? See Step 5 of this instructions.


Maya type validation in the MayaInterval arguments?

Should we force validation of the "Maya" type check on the arguments passed to the "MayaInterval" class? If yes, I am up for this task.

Sure!


Compile Docker image and Linux binary on GitHub actions when a new vX.X.X tag is created

Assign to me please. 🙏

I am not a Docker expert, but before calling docker push, we'll need to do docker login, I guess. Could you please provide me with command we'll use to login? I assume there will be some token used. You need to save this token into secrets. And then tell me the name of the secret, so I can access it from the workflow.

Let's hold off pushing the image for now and just store the binary. I wish to re-work the docker image and repo.


Traceback errors on export

Describe the bug
A clear and concise description of what the bug is.
Trying to export a modified Asobo aircraft is giving me traceback errors:

I've recently reinstalled Blender 2.92 with Windows 10, no other MSFS stuff active (getting activation issues with MSFSToolkit too). I deleted Blender originally due to this issue too, hoping it would resolve on reinstall.
To Reproduce
Steps to reproduce the behavior:
As always:

Select all items to export
choose Export

Select the file to export
Get the error posted above

Expected behavior
As always, a black loading square would appear, and the .gltf and .bin files would be created.
Screenshots
If applicable, add screenshots to help explain your problem.
Embedded above
.blend file/ .gltf
If applicable, a zipped folder containing a .blend file for exporter issues and a .gltf file for importer issues. Please take care to include all resources, such as textures in the .zip file.
Version

OS: [e.g. macOS, linux] Win 10
Blender Version [e.g. 2.82, 2.83, or build date] 2.92

Additional context
Add any other context about the problem here.

If there's anything that seems obvious that I've missed so far, please let me know and I'll remove the issue. Been stuck with this since SU5, and I understand the discord group isn't involved anymore (plus that channel is locked).

im getting the same Error

Can you show your addon settings?

Can you show your addon settings?

I meant in Preferences -> Add-ons

This?

I meant in Preferences -> Add-ons

Now im getting this errror

One of your meshes has a very high number of vertices. The exporter is trying to split the mesh into separate primitives, however, there are cases where it is unable to, such as yours. You're going to need to locate any large meshes and either reduce the vert count or split them.


Reduce dependencies of datafusion-sql crate

Which issue does this PR close?
This doesn't close, but contributes to the changes being worked on in #1750.
Rationale for this change
By disabling features and removing unused dependencies the build time of those crates is reduced.
What changes are included in this PR?
This PR removes some unused direct dependencies of the sql, common and expr crates, and disables unused features.
Are there any user-facing changes?
No

Codecov Report

Merging #3566 (f522f88) into master (81b5794) will increase coverage by 0.10%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3566      +/-   ##
==========================================
+ Coverage   85.81%   85.91%   +0.10%
==========================================
  Files         300      301       +1
  Lines       55548    56218     +670
==========================================
+ Hits        47666    48301     +635
- Misses       7882     7917      +35

Impacted Files
Coverage Δ

datafusion/sql/src/planner.rs
81.25% <100.00%> (ø)

datafusion/optimizer/src/type_coercion.rs
92.18% <0.00%> (-1.38%)
:arrow_down:

...sion/core/src/physical_plan/file_format/parquet.rs
94.65% <0.00%> (-0.25%)
:arrow_down:

datafusion/expr/src/logical_plan/plan.rs
77.75% <0.00%> (-0.17%)
:arrow_down:

datafusion/core/tests/sql/predicates.rs
100.00% <0.00%> (ø)

...tafusion/core/src/physical_plan/file_format/mod.rs
96.94% <0.00%> (ø)

datafusion/optimizer/src/reduce_cross_join.rs
94.54% <0.00%> (ø)

datafusion/core/src/execution/context.rs
79.33% <0.00%> (+0.02%)
:arrow_up:

datafusion/optimizer/src/projection_push_down.rs
98.11% <0.00%> (+0.04%)
:arrow_up:

datafusion/common/src/scalar.rs
85.18% <0.00%> (+0.06%)
:arrow_up:

... and 6 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks @mbrobbel


About blackbox stateful register if I want to convert p4 Tofino to Bmv2.

Excuse me, I find that A2TP just run in the Tofino, but not all person has the Tofino switch. So if we want to replicate experiment, we can
just run code in the  Bmv2, but the blackbox stateful register don't exist in the Bmv2. Can you help me? a.)Tell we ablout the blackbox stateful register constructs in the Tofino P4_14 compiler, what it is doing in the general. Or b.)Whether I have a chance to go to your lab，and learn more about the A2TP.
Thanks a lot.

Do you solve it？

Sorry for the late reply. In Tofino P4-14, blackbox stateful is used to access and update registers. I believe the same logic can be implemented in Bmv2. You could join the Intel Connectivity Research Program (ICRP) to access the official document, where you can find detailed information about tofino stateful processing. Due to confidentiality agreements from ICRP, I cannot publicly show you the document.


Cannot load elmo (trained on a new corpus) in allennlp

Hi Matthew, thanks for your code.
I followed the 5-step Training a biLM on a new corpus and passed the first 4 steps. But in the last step, I had a problem when loading elmo trained on my own corpus using the code like here (Using ELMo programmatically)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/allennlp/modules/elmo.py", line 99, in __init__
    vocab_to_cache=vocab_to_cache)
  File "/usr/local/lib/python3.6/dist-packages/allennlp/modules/elmo.py", line 528, in __init__
    self._elmo_lstm.load_weights(weight_file)
  File "/usr/local/lib/python3.6/dist-packages/allennlp/modules/elmo_lstm.py", line 299, in load_weights
    proj_weights = numpy.transpose(dataset['W_P_0'][...])
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/group.py", line 177, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'W_P_0' doesn't exist)"

and the corresponding options.json I used for training and dumping is
{"bidirectional": true,
     "char_cnn": {"activation": "relu",
                  "embedding": {"dim": 16},
                  "filters": [[1, 32], [2, 16], [3, 16], [4, 128], [5, 128], [6, 256], [7, 256]],
                  "max_characters_per_token": 50,
                  "n_characters": 261,
                  "n_highway": 2},
     "dropout": 0.1,
     "lstm": {"cell_clip": 3,
              "dim": 512, "n_layers": 2,
              "proj_clip": 3,
              "projection_dim": 512,
              "use_skip_connections": true},
     "all_clip_norm_val": 10.0,
     "n_epochs": 40,
     "n_train_tokens": 23730468,
     "batch_size": 2,
     "n_tokens_vocab": 30000,
     "unroll_steps": 40,
     "n_negative_samples_batch": 128}

I don't know where should I add the W_P_0 mentioned in the error.
How can I fix this?
best,
Zhenghui

It seems that it works if I add "projection_after_highway": true in the options.
so now it is
{"bidirectional": true,
     "char_cnn": {"activation": "relu",
                  "embedding": {"dim": 16},
                  "projection_after_highway": true
                  "filters": [[1, 32], [2, 16], [3, 16], [4, 128], [5, 128], [6, 256], [7, 256]],
                  "max_characters_per_token": 50,
                  "n_characters": 261,
                  "n_highway": 2},
     "dropout": 0.1,
     "lstm": {"cell_clip": 3,
              "dim": 512, "n_layers": 2,
              "proj_clip": 3,
              "projection_dim": 512,
              "use_skip_connections": true},
     "all_clip_norm_val": 10.0,
     "n_epochs": 40,
     "n_train_tokens": 23730468,
     "batch_size": 2,
     "n_tokens_vocab": 30000,
     "unroll_steps": 40,
     "n_negative_samples_batch": 128}

Is this the correct solution?
I find this settings also in options But this line (projection_after_highway": true) is now included explicitly in train_elmo.py, maybe it would be better to include it in train_elmo.py？


check for columtype is wrong

Tag: V1.3.2
core/fcpayone_events.php:578
Should the expected value not be char32 instead of int11? Currently, as far as I can see, it does not perform the migration because of this and the column remains an int11. This caused problems with the status.php.
Greetings,
Christoph

For reference: https://github.com/PAYONE-GmbH/oxid-6/blob/30d3cad7c1365d8c3fc7b5bda3281c35ef9e9f78/core/fcpayone_events.php#L578
In a test shop in our system, this also looks rather suspicious (which might be because of our outrageously long order numbers, but still):

I'll talk to Dev about it.


Url-encode paths dont't work for both mdlink and wikilink.

Hello there!
Some markdown editor may  url-encode some parts of md link or wiki link, this behavior breaks the marksman lsp functionality(defination, references, etc.).
Test case
[test](#Test%20test)   --  not work

[[##Test%20test]]  --  not work

[[##Test test]]  --  work

marksman version
2023-06-01

Thanks for reporting @EvanLind ! #216 should fix the issue.


[idea] Frame step feature

Hi!
Right now the Frame step works like a single click - single action
Is it possible to constantly execute this button hander until the button being pressed down?

once the button pressed up - stop the handler callings...

I looked at adding this, but it just behaves as if the play rate is x2. It might be possible to add in a timer, but then we are effectively duplicating the play rate that the engine provides. So unfortunately I don't think its something I will add for now.


add a few buttons

name: Pull request
about: Create a pull request to contribute in making Pock amazing!
title: add single volume toggle and single brightness toggle button
labels: add single volume toggle and single brightness toggle button
assignees: ''

Describe the Pull request

add single volume toggle and single brightness toggle button.
volume toggle button's icon can changes depends on current system volume.
add screensaver button #256
fix #273

Screenshots

Pull request type

[x] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Tested?
Yes
Test Configuration:

Hardware:      [MacBook Pro (13-inch, 2018, Four Thunderbolt 3 Ports)]
macOS Version: [10.15.1]
Xcode version: [11.2]

Checklist
Use this list to keep track of your progress:

[X] I have performed a self-review of my own code
[X] I have commented my code, particularly in hard-to-understand areas
[X] My changes generate no new warnings
[X] I have added tests that prove my fix is effective or that my feature works

add system-like mute icon
unmuted：

muted：

add high resolution icons
before：

after：

add screensaver button
Demo Here

@pigigaldi
When system is muted, NSSound.isMuted returns false. When the system is unmuted, it returns true.
I think it returns last value and is not writable.
I don’t know how to detect if the system is muted or unmuted, so that I don’t know which icon the volume button should display.
Do you have any idea?

@L1cardo I think your pull request is a great addition and I would like to help. If I understand you correctly, you have no reliable way of how to figure out if the system is muted or not?
You could use this code snippet that uses CoreAudio to achieve this:
import CoreAudio

    var isAudioMuted: Bool {
         var defaultHardwareAddress = AudioObjectPropertyAddress(
            mSelector: AudioObjectPropertySelector(kAudioHardwarePropertyDefaultOutputDevice),
            mScope: AudioObjectPropertyScope(kAudioObjectPropertyScopeGlobal),
            mElement: AudioObjectPropertyElement(kAudioObjectPropertyElementMaster))

         var defaultAudioHardwareID: AudioDeviceID = 0
         var defaultHardwarePropertySize = UInt32(MemoryLayout<uint32>.size)

        _ = AudioObjectGetPropertyData(
            AudioObjectID(kAudioObjectSystemObject),
            &defaultHardwareAddress, 0 , nil,
            &defaultHardwarePropertySize,
            &defaultAudioHardwareID)

         // kAudioDevicePropertyMute: A UInt32 where a value of 1 means that mute is enabled making that element inaudible.
         var muteStatusAddress = AudioObjectPropertyAddress(
            mSelector: AudioObjectPropertySelector(kAudioDevicePropertyMute),
            mScope: AudioObjectPropertyScope(kAudioObjectPropertyScopeOutput),
            mElement: AudioObjectPropertyElement(kAudioObjectPropertyElementMaster))

        var isMuted: uint32 = 0
        var muteStatusPropertySize = UInt32(MemoryLayout<uint32>.size)

         _ = AudioObjectGetPropertyData(
            defaultAudioHardwareID,
            &muteStatusAddress, 0, nil,
            &muteStatusPropertySize,
            &isMuted)

         print("Mute Status: \((isMuted == 1) ? "muted" : "unmuted")")

        return isMuted == 1
    }

@andreashanft Hi! Thanks for your help, I will give it a shot this weekend. But now I should  probably working on my exam 😭

@andreashanft I tested the code that you offer, but it was still the same problem. What I mean is that , NSSound.isMuted does not response immediately , it always returns last status of isMutued, which is when system is muted, NSSound.isMuted returns false and when system is unmuted, it returns true. So the mute image that is on the touchbar is based on the last status of the isMutued, and always not correct .

change screensaver button icon to the system-like icon

change lock screen icon to the system-like icon

@pigigaldi
When system is muted, NSSound.isMuted returns false. When the system is unmuted, it returns true.
I think it returns last value and is not writable.
I don’t know how to detect if the system is muted or unmuted, so that I don’t know which icon the volume button should display.
Do you have any idea?

I think I have fixed it

icons of volume mute item and volume toggle item now sync with system volume settings

add "Do not Disturb" button
You can check it out here

add missing localizations
But I have a question:
Can we have system folders names localized that are shown on Dock ?


Poison Expert did not give me proficiency in the Poisoners kit.

I had my rogue take the feat Poison Expert and it did not give him proficiency in the poisoner's kit.
Poison Expert issue.zip

fixed in dev. thanks.


SCP-3267 setup files for marlowe cli wrapper and tests

We're going to create our own marlowe cli DSL and use that to write tests. Closing this PR


Test helper 'fillInFroalaEditor' not working

I'm tring to use the helper fillInFroalaEditor in my acceptance test but isn't working correctly. The helper fill in the froala instance with the text that i have passed, but the content property is not updated with the text unless i force the test to click in something else like a button to take the focus off the froala editor.

Can you provide a snippet of code which you are trying to accomplish? The helper uses a Froala function to set the content, so I'm surprised the editor wouldn't pick up that change immediately.

yea, we can see the same issue
visit("/page/" + category.id);
    andThen(function () {
      fillInFroalaEditor('#page .froala-editor-container', '
asdasd
');
      andThen(() => {
         assert.equal(find(`.details .title}')`).text().trim(), `${title} *`);
    });

When the content property is updated we append a * to a tab and the assertion above checks if it's appended. So the content property is not updated with the fillIn helper.
even with a simple test it doesnt update
in template the follwing above the editor:

{{model.content}}

fillInFroalaEditor('#knb-entry-custom-information-form .froala-editor-container', '
asdasd
');
      andThen(function () {
        assert.equal(find('#foo').text().trim(), "asdasd");
      });

Ah, so you are expecting the update event to fire too (not just checking the DOM). I'm thinking it gets back to the fact that html.set does not trigger the contentChanged event. I just pushed a change that will force the event to fire, triggering your typical update event.

works perfectly for me :) i hope for @raphaelns-developer  too

Great! I have a few other things to update here and then I'll publish a new version.

Great! I think this is the solution, but i will test it tomorrow

@raphaelns-developer Sorry for the delay, but I'm just unsure and tried to do some research but came up empty. Have you tried using it? If anything, I wonder if the import process would be different for acceptance vs integration.

@Panman8201 Yes, i already tried, but in the helper you use the find helper and i know no why it only works in acceptance tests. Maybe if you change the find to JQuery it will work in any test.

Closing this as Version 2.7.5 includes this fix.
@raphaelns-developer if you want to continue exploring the helper in integration tests let's open another issue.


Using secure copy to import IOS

I am a student at ASU and I am working on my capstone project.
I am trying to write a python program that will update the the Cisco IOS via netmiko.
I do have ip scp server enabled on my 2960 Cisco Switch.
When I run the program I get the following error code:
File "C:\Users\stscs\AppData\Local\Programs\Python\Python38\lib\site-packages\netmiko\scp_handler.py", line 360, in transfer_file
self.put_file()
File "C:\Users\stscs\AppData\Local\Programs\Python\Python38\lib\site-packages\netmiko\scp_handler.py", line 373, in put_file
self.scp_conn.scp_transfer_file(self.source_file, destination)
File "C:\Users\stscs\AppData\Local\Programs\Python\Python38\lib\site-packages\netmiko\scp_handler.py", line 46, in scp_transfer_file
self.scp_client.put(source_file, dest_file)
File "C:\Users\stscs\AppData\Local\Programs\Python\Python38\lib\site-packages\scp.py", line 158, in put
self._recv_confirm()
File "C:\Users\stscs\AppData\Local\Programs\Python\Python38\lib\site-packages\scp.py", line 363, in _recv_confirm
raise SCPException(asunicode(msg[1:]))
scp.SCPException: Privilege denied.

You need privilege level 15 access on the account you are using to SCP (direct i.e. you can't require an enable call since there is no way to do this on the Secure Copy channel).

Makes sense: so I wrote the following code into my function to hopefully configure SCP:
ssh = ConnectHandler(**conn)
ssh.enable()
ssh.send_command('aaa  authentication login default local')
ssh.send_command('username easypair privilege 15 password cisco')
ssh.send_command('ip scp server enable')

remaining code to import IOS:
transfers IOS to Cisco device and will overwrite the current IOS
    file_transfer(ssh,
    source_file = IOS,
    dest_file = IOS,
    file_system = 'flash:',
    direction = 'put',
    disable_md5= True,
    overwrite_file = False,)
    ssh.disconnect()
    messagebox.showinfo('Info', 'Process is Complete')


fix missing pass options down to getSingleSelector.

fixes #15

Thanks for spotting - options are now properly passed.


refactor(groupBy): reduce the size of the implementation

Refactor to make it smaller
Replaced a faulty test with a test that tests the same thing. Verified the behavior has not changed since 6.x.

BEFORE: 1.99KB

AFTER: 1012B (50% of the original size)

Closing in favor of #5729 ... (Actually merging these commits into that one then working on them some more)


[doc] Describe countermeasure keys in comportability spec

I should note that I think there are some slightly dubious entries here. For example, I can't see a situation where the (open source) OpenTitan project is going to use GLITCH_DETECT. But this PR is intended to match the current list of countermeasure names (as used in the product-specific Google Sheet). I propose any bikeshedding about names should be in a separate PR :-D
Closes #9985.

looks good to me, thanks for adding this.

@rswarbrick i'm going to merge this first :)

btw, this looks good to me.
When I was going through the first D2S review process earlier today, I think I discovered one thing missing (@msfschaffner and I chatted about this) - I think we need a REGWEN countermeasure (holds value until next reset, I think only applies to CONFIG items), since it's a way we lock behaviors as the part advances through its boot process.

also, REDUN can apply to CTR assets, I believe

also, REDUN can apply to CTR assets, I believe

Yep, definitely. That column is supposed to be a "commonly used with" list, so maybe it doesn't matter too much? Also I think I was just copying the list from the spreadsheet, so I take no responsibility for omissions :-D

btw, this looks good to me.
When I was going through the first D2S review process earlier today, I think I discovered one thing missing (@msfschaffner and I chatted about this) - I think we need a REGWEN countermeasure (holds value until next reset, I think only applies to CONFIG items), since it's a way we lock behaviors as the part advances through its boot process.

Indeed, we should add REGWEN to CM_TYPE and I think we also identifyied RNG as an ASSET in the entropy_src D2S review today.

also, REDUN can apply to CTR assets, I believe

Yep, definitely. That column is supposed to be a "commonly used with" list, so maybe it doesn't matter too much? Also I think I was just copying the list from the spreadsheet, so I take no responsibility for omissions :-D

Hah, I didn't realize you were transcribing, my apologies!
In any case, we should try to give good suggestions so adding these things from the ongoing D2S reviews will be useful.


TypedefType should contain a name and another Type

Invalid. xrg.types.TypedefType satisfies the public api for TypeBase. It's a bit confusing because the declaration class xrg.declarations.TypedefItem does carry both a name and a type.


Remix Router V2

Working draft of documentation pages using directus as a cms

New env variable:
VERCEL_CLI_VERSION =<EMAIL_ADDRESS>Vercel CLI has problems deploying remix with V2 routing and the last known version to my knowledge is 28.15.3
If we're deploying to PROD we need to change the CLI to this version inside vercel settings, right now I'm only using it for dev/preview.


add Debug and Set at startup project commands

to F# project explorer, if project is a console app

Debug will run the debugger of selected project. no launch.json needed. this just fix a bug in the previous @Krzysztof-Cieslak implementation

NOTE the gif is not updated, the Debug is below Run in the context menu

Set at startup project update launch.json or create it if not exists

NOTE for .NET full (`"type": "clr") the debugging on windows works like before (for old or new sdk). but project must be configured as x64 and portable pdb
<PlatformTarget>x64</PlatformTarget>
<DebugType>portable</DebugType>

Set at startup project should be called Generate run and debug config - I have something else in mind regarding setting startup project. Tasks for restore & build should be generate


docs(changelogs): fix changelog format

Codecov Report
Base: 90.44% // Head: 90.44% // No change to project coverage :thumbsup:

Coverage data is based on head (acd668b) compared to base (3cb4b18).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1163   +/-   ##
========================================
  Coverage    90.44%   90.44%
========================================
  Files          181      181
  Lines         5516     5516
========================================
  Hits          4989     4989
  Misses         527      527

Flag
Coverage Δ

py3.8-ubuntu-latest-pandas
90.44% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Create workflow actions for automated releasing

New release and snapshots actions for automated publishing to Maven Central. Fulfills https://github.com/getditto/DittoAndroidTools/issues/80
Release workflow triggers on new release publish in GitHub - all projects with the same version deployed at the same time - so we can keep consistency.  Signing is automated, using GitHub's secrets.
Snapshots workflow currently doesn't work but would ideally trigger nightly. Leaving the script in there to eventually debug & fix.
Let me know if it makes sense - happy to walk you folks through it & answer any Qs.
And most importantly thanks Raj for your help looking at error logs!!!

Sorry if this is a silly question as I'm not that familiar with GHA. Can anyone with a GH account run the GHA, or do we need to setup permissions so that only certain people/level of access can publish?

I think we should add either a RELEASING.md or something in Notion to describe how to release to describe the workflow for updating the versions then publishing etc.

Sorry if this is a silly question as I'm not that familiar with GHA. Can anyone with a GH account run the GHA, or do we need to setup permissions so that only certain people/level of access can publish?

At the moment, anyone with write access to the repo can publish a release which will trigger the release action. Seemed like a sensible default option.
If we need to, we can change permissions / triggers so that only members of certain groups can do it. Happy to hear others' thoughts?

I think we should add either a RELEASING.md or something in Notion to describe how to release to describe the workflow for updating the versions then publishing etc.

Great idea - I'll write it up this morning. Can draft it in notion then either move to the repo or keep there.

I documented everything - the process and the scripts.
I think this should cover everything: https://www.notion.so/getditto/Publishing-Automation-f3ff687ee9f24e059bbc85af716f2d3d

@zmarkan Thanks for documenting! I think that is the right place to leave those docs given this is a public repo

Nice!

So do we just merge now it's been approved? Or is there a step in the flow that I'm missing?

So do we just merge now it's been approved? Or is there a step in the flow that I'm missing?

Yes you can merge!


Rule re render

@TimGuiteDiamond I believe that this should prevent the rule re-renders that you were seeing, but I would like you to confirm.

Yep this seems to do what it says on the tin, happy to merge


issue #23: now scans also jars without .jar extension

Hi
This is my proposal of implementing issue #23. I created already one pull request yesterday but I wanted to change something. I'm new to github and I had problem with adding new changes to already opened pull request. That's why I do a new one.
Please have a look at this

@jfarcand Note that in your implementation the InputStream is not closed when an Exception is thrown.
(okay, i know i am very late! :) )

@rmuller Thanks, fixed by 6e4a3bf


[SPARK-32850][CORE][K8S] Simplify the RPC message flow of decommission

What changes were proposed in this pull request?

This PR cleans up the RPC message flow among the multiple decommission use cases, it includes changes:

Keep Worker's decommission status be consistent between the case where decommission starts from Worker and the case where decommission starts from the MasterWebUI: sending DecommissionWorker from Master to Worker in the latter case.

Change from two-way communication to one-way communication when notifying decommission between driver and executor: it's obviously unnecessary for the executor to acknowledge the decommission status to the driver since the decommission request is from the driver. And it's same in reverse.

Only send one message instead of two(DecommissionSelf/DecommissionBlockManager) when decommission the executor: executor and BlockManager are in the same JVM.

Clean up codes around here.

Why are the changes needed?

Before:

After:

(Note the diagrams only counts those RPC calls that needed to go through the network. Local RPC calls are not counted here.)
After this change, We reduced 6 original RPC calls and added one more RPC call for keeping the consistent decommission status for the Worker. And the RPC flow becomes more clear.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Updated existing tests.

Bring this back since it isn't the original commit that breaks the K8S test. As the PR https://github.com/apache/spark/pull/29751 merged before this already failed the K8S tests

Test build #128934 has started for PR 29817 at commit 5ca0fe8.

@holdenk, seems like the test failure wasn't caused by this PR. Dose your -1 at here still stand?
cc @dongjoon-hyun as well per #29408.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33556/

The -1 is only on the PR that I mentioned it on (the refactoring with many sub classes) which still stands. I would like to review this PR more though since I think we probably need better test coverage of this change than it originally had. Sound good? Thanks for checking in about that :)

Test build #128935 has started for PR 29817 at commit 15f6085.

Got it about -1 but what about we pushing this as is, and work on the test as followups? It's a bit odds that we reverted it for the reason this PR didn't cause, and ask more things before merging it in.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33556/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33556/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33557/
Test FAILed.

I think we should not commit this with the K8s test being broken. It’s in the same chunk of code and changes the logging string (although another PR also changed that string first too?). I do not believe this PR was appropriately tested when first merger given it changed decommissioning messages and did not run the decommission tests.
For clarity: if you want to fix the tests in a separate PR that’s ok with me, but I would prefer not to commit this without passing integration testing.

Test build #128934 has finished for PR 29817 at commit 5ca0fe8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128934/
Test PASSed.

Test build #128935 has finished for PR 29817 at commit 15f6085.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class DecommissionWorkers(ids: Seq[String]) extends DeployMessage
  case class WorkerDecommissioning(id: String, workerRef: RpcEndpointRef) extends DeployMessage
  case class ExecutorDecommissioning(executorId: String) extends CoarseGrainedClusterMessage

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128935/
Test PASSed.

I think we should not commit this with the K8s test being broken.

We don't. That's also why I added [K8S] tag in the PR title. And feel free to leave comments, I can address them in followups.

@holdenk, why don't you take a look for a test failure since it blocks all changes in decommission in k8s, and you were involved mainly in the development there?

Can we find out which commit caused the test failure in the first place? We should either revert that commit, or fix it soon, as the test failure blocks others.
Since this PR is resubmitted (although the revert is not necessary now given the test failure was already there), I think it's a good chance for @holdenk to take a closer look before re-merging. And I agree with @holdenk that we can't merge a PR when the related test is already broken. We should fix that first. @holdenk can you give some hints about it? I took a look at https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33556/ , but I don't even see how the test failed. The output is very different from normal Spark tests.

I really like the idea of simplifying the RPC message flow, thanks for taking this on @Ngone51 and I'm sorry the code here is so brittle to these types of changes (the K8s integration tests are kind of limited).

but that's only half true, this PR just broke the test some more.

@holdenk, you're kidding right? There was only one test failure that was not caused by this PR in K8S tests. That test was just fixed by you. How come this PR broke more tests? Can you be more explicit on that? Which tests were more broken, and how did you test?

I’m not joking. The test has multiple conditions. Another PR broke one of the conditions. This PR in its original broke another one of the conditions. It was reported in the same test because K8s integration tests focus on the integration so they cover multiple pieces. If you want I’m happy to do a video call (and I can be flex on my timezone if that’s a constraint) and screen share so we can discuss the details.

If it helps I called out the part of the PR I believe most likely responsible for that during my code review already.

This PR in its original broke another one of the conditions.

Except for the breaking of the log(like you just fixed), what other conditions this PR breaks?

That being said I still have concerns this PR is not sufficiently tested, can you add some more tests for the new flows you've introduced?

There's only one new flow that is from Master to Worker. I can update the existing test by verifying Worker's decommission status... What kind of other concerns do you have? Could you elaborate more? So I can improve the PR accordingly.

I’m mostly concerned with the change around how the storage decommissioning is being done now, I’d like to see some tests that the flow from the master to the worker results in storage decommissioning.

Also there are multiple log conditions, this PR broke one of them. Another PR had broken another one.

@Ngone51, shall we add a test https://github.com/apache/spark/pull/29817#issuecomment-698438316 and fix the test as requested? Seems like otherwise good to go.

Test build #129172 has started for PR 29817 at commit 23bfdf5.

@holdenk Thanks for the review! I've addressed your most comments. And tests are updated and added in fa04b493324ea52698bed7ce15795530052b37be and
9d0f36d73e5750267ec2f33e7d964fc09778767f .

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33787/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33787/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33787/
Test FAILed.

Test build #129172 has finished for PR 29817 at commit 23bfdf5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129172/
Test PASSed.

Test build #129182 has started for PR 29817 at commit e3ca2af.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33795/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33795/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33795/
Test FAILed.

Test build #129182 has finished for PR 29817 at commit e3ca2af.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129182/
Test FAILed.

retest this please

Test build #129198 has started for PR 29817 at commit e3ca2af.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33813/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33813/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33813/
Test FAILed.

Tests are still failing @Ngone51 happy to take another look once the K8s integration tests are passing.

Test build #129198 has finished for PR 29817 at commit e3ca2af.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129198/
Test PASSed.

Test build #129277 has started for PR 29817 at commit af9dbdd.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33894/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33894/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33894/
Test FAILed.

Test build #129277 has finished for PR 29817 at commit af9dbdd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129277/
Test PASSed.

@holdenk Sorry for the delay. I had taken some days off recently. I'm addressing your comments right now and debugging the integration test(the decommissioned executor doesn't migrate shuffle blocks as expected, so I'm still investigating it.)
Thank you for the further review :)

Taking days off is healthy and great! We're not in a rush on this so please take your time. My offer of help with debugging tests was not intended to make you feel like you needed to work on this right away, just I've been in the situation where I'm stuck on a test failure I can't figure out in the past. Let me know when you want me to take another look :)

Test build #129640 has started for PR 29817 at commit 0064ba8.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34244/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34244/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34244/
Test PASSed.

cc @holdenk for taking another look. Thanks!

Test build #129640 has finished for PR 29817 at commit 0064ba8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129640/
Test PASSed.

Test build #129652 has started for PR 29817 at commit 17bbbd9.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34256/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34256/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34256/
Test PASSed.

Test build #129652 has finished for PR 29817 at commit 17bbbd9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129652/
Test PASSed.

kindly ping @holdenk

@holdenk take a look?

Let me merge this in few days if there are no more comments assuming lazy consensus.

@holdenk ?

Retest this please.

Test build #130119 has started for PR 29817 at commit 17bbbd9.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34727/
Test FAILed.

Test build #130119 has finished for PR 29817 at commit 17bbbd9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130119/
Test FAILed.

Test build #130130 has started for PR 29817 at commit e625051.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34737/

Test build #130130 has finished for PR 29817 at commit e625051.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130130/
Test FAILed.

retest this please

Test build #130140 has started for PR 29817 at commit e625051.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34737/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34737/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34747/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34747/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34747/
Test FAILed.

retest this please

Test build #130148 has started for PR 29817 at commit e625051.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34755/

Test build #130140 has finished for PR 29817 at commit e625051.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130140/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34755/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34755/
Test FAILed.

Test build #130148 has finished for PR 29817 at commit e625051.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130148/
Test FAILed.

retest this please

Test build #130159 has started for PR 29817 at commit e625051.

Test build #130163 has started for PR 29817 at commit 3c1e033.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34766/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34770/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34766/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34766/
Test FAILed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34770/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34770/
Test PASSed.

Test build #130159 has finished for PR 29817 at commit e625051.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130159/
Test PASSed.

Test build #130163 has finished for PR 29817 at commit 3c1e033.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130163/
Test PASSed.

Merged to master.

thanks all!!


replace "forever" with pm2

currently, to keep the magibot process running 24/7 even if for some reason it crashes, we use forever
its not being actively maintained well anymore, and they suggest switching to an alternative
https://pm2.keymetrics.io/ looks like a good option imo

https://discordjs.guide/improving-dev-environment/pm2.html#managing-your-bot-process-with-pm2

not needed anymore, as its now deployed in a container that should take care of keeping itself alive


Table resizing

Hi, I've been using react-bootstrap-table and it's awesome.
However, there is a small bug annoying me. When I resize the browser window the table header and table body are scaled independently which results in a shift between them. I think this happens due to the line breaking when column width is too small for the column name. Here is a screenshoot:

You can see Inserted rows is break into two lines to fit the screen width.
It is manifested on Google Chrome and Firefox.
Also sometimes even when the screen is full width, and no line breaking took place, this kind of shift between table header and table body occurrs.

@marin101, you forgot to add css of react-bootstrap-table, check https://github.com/AllenFang/react-bootstrap-table#cimport-css.
Let me know if the problem still remain after you add css.  Thanks!

 Sorry for the fake issue

that's ok :)

Hi,
I'm just starting to use react-bootstrap-table, i want to be able to resize my columns, do you have any ideas on how to proceed?
Thanks

@yha1990, there's a PR already fix it but need some time, I've no schedule now, sorry.

@AllenFang  thank you for your response :)

follow-up on this issue ?


News - text outside of box

Going to handle the checksums (or any non-breaking long string) without using CSS - this way, we won't need to sacrifice the whitespace breaking for the entry text.


More fixes

https://github.com/rancher/rancher/issues/6626
https://github.com/rancher/rancher/issues/6595
https://github.com/rancher/rancher/issues/6580
https://github.com/rancher/rancher/issues/6690

https://github.com/rancher/rancher/issues/388


[Draft] SecurityHeaders for React client server

Note: Content-Security-Policy can have the line feeds removed (aka combine into a single, long line). It's just more human readable in the format below:
"SecurityHeaders": {
 "Content-Security-Policy": "
    default-src 'self';
    script-src 'self'
      https://hss-prime.okta.com
      https://global.oktacdn.com
      https://www.google-analytics.com
      https://*.in.applicationinsights.azure.com;
    style-src 'self' 'unsafe-inline'
      https://global.oktacdn.com
      https://cdnjs.cloudflare.com;
    frame-src 'self' https://hhs-prime.okta.com;
    img-src 'self' https://hhs-prime.okta.com https://reportstream.cdc.gov data: ;
    connect-src 'self'
      https://www.google-analytics.com
      https://*.in.applicationinsights.azure.com
      https://hhs-prime.okta.com
      https://staging.reportstream.cdc.gov/api/
      https://staging.prime.cdc.gov/api/ ;
    ",

    "X-Content-Type-Options": "nosniff",
    "X-Frame-Options": "DENY",
    "X-Permitted-Cross-Domain-Policies": "master-only",
    "X-Xss-Protection": "1; mode=block;",

    "Referrer-Policy": "origin-when-cross-origin",
    "Strict-Transport-Security": "max-age=31536000; includeSubDomains; preload",

    "x-azure-ref": null,
    "x-ms-request-id": null,
    "x-ms-version": null,
  },

The last few remove the header that Azure adds.
Other possible items for Content-Security-Policy? Not sure how much security they actually add based on defaults.
  script-src 'self'
    'nonce-{nonce}'
    https://www.google-analytics.com/analytics.js;

  font-src 'self' data: ;

  object-src 'none';

  base-uri staging.reportstream.cdc.gov 'self';

  frame-src 'self';

  frame-ancestors 'none';

  form-action 'self';

  manifest-src 'self';

  media-src 'self';

  worker-src 'none';

A few things:

Are we planning on using these rules as-is for all environments? Or are we going to templatize/customize per environment? For example this would impact connect-src which only explicitly mentions staging.
Are we or any of our dependent services using web workers? (see: child-src
Based on my understanding we don't need to explicitly specify rules that don't modify what's in default-src -- e.g., media-src 'self'

Just some implementation notes:

There is no way to set HTTP headers on the storage account (Azure docs here)
There's a way to do this via the Front Door configuration (Azure docs here) However, note that Terraform does not yet support this -- see this issue -- so we'd have to add manually. Keep an eye on the issue comments and workarounds for ensuring tf doesn't remove manual configuration. Not sure if it's necessary, but good to know.
We can probably also do this in Akamai, but the fact that this might be done in the future in terraform config makes me lean toward that.

doing this manually via the Front Door config makes sense to me! maybe we should just have a Readme somewhere in our repo with the configurations so that we at least have version history of any changes? we'd have to make sure the Readme stays up to date, but I don't imagine changes to SecurityHeaders will happen too frequently

We may make updates to this later, but the basics are working now in the Front Door.


Support nodejs 12

Current build errors:
../src/node_snap7_client.cpp: In static member function ‘static void node_snap7::S7Client::Init(Nan::ADDON_REGISTER_FUNCTION_ARGS_TYPE)’:
../src/node_snap7_client.cpp:697:43: error: no matching function for call to ‘v8::FunctionTemplate::GetFunction()’
   Nan::Set(target, name, tpl->GetFunction());
                                           ^
../src/node_snap7_client.cpp:697:43: note: candidate is:
In file included from /home/travis/.node-gyp/12.1.0/include/node/node.h:63:0,
                 from ../src/node_snap7_client.h:10,
                 from ../src/node_snap7_client.cpp:6:
/home/travis/.node-gyp/12.1.0/include/node/v8.h:5947:46: note: v8::MaybeLocal<v8::Function> v8::FunctionTemplate::GetFunction(v8::Local<v8::Context>)
   V8_WARN_UNUSED_RESULT MaybeLocal<Function> GetFunction(
                                              ^
/home/travis/.node-gyp/12.1.0/include/node/v8.h:5947:46: note:   candidate expects 1 argument, 0 provided
../src/node_snap7_client.cpp: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE node_snap7::S7Client::New(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/node_snap7_client.cpp:710:53: error: no matching function for call to ‘v8::FunctionTemplate::GetFunction()’
     Nan::NewInstance(constructorHandle->GetFunction()).ToLocalChecked());
                                                     ^
../src/node_snap7_client.cpp:710:53: note: candidate is:
In file included from /home/travis/.node-gyp/12.1.0/include/node/node.h:63:0,
                 from ../src/node_snap7_client.h:10,
                 from ../src/node_snap7_client.cpp:6:
/home/travis/.node-gyp/12.1.0/include/node/v8.h:5947:46: note: v8::MaybeLocal<v8::Function> v8::FunctionTemplate::GetFunction(v8::Local<v8::Context>)
   V8_WARN_UNUSED_RESULT MaybeLocal<Function> GetFunction(
                                              ^
/home/travis/.node-gyp/12.1.0/include/node/v8.h:5947:46: note:   candidate expects 1 argument, 0 provided
make: *** [Release/obj.target/node_snap7/src/node_snap7_client.o] Error 1

Hi @Apollon77 . I'll have a look in the next days. Shouldn't be hard to fix. I'm currently rewriting this module for NAPI in my rare spare-time, so in the future major version changes will be less error-prone.

Thank you very much. I know that these changes are a pain ... I also have one „binary package“ Modul to maintain :-(

@mathiask88 FYI (because I got a build failure notification from AppVeyor for this project for some reason): node 12 is not supported yet, but there's a workaround: https://github.com/appveyor/ci/issues/2921#issuecomment-486727727

@vweevers It is no real "workaround" Update-NodeJsInstallation did always pull a fresh version and Install-Product selects a preinstalled version. The preinstalled versions lag, but I have the time to wait. However thanks for being attentive! :) I don't know why you got the notifications. Maybe because prebuild used my appveyor account in the past.

I decided for ourt perojects to switch to the "workaround method". I'm better with "latest from a certain version chain" then some old stuff, so I can tell the customers "update your node" first ...

AppVeyor now has node v12 in their build images, but to be future-proof see https://github.com/mathiask88/node-snap7/commit/6738c971fc5af547f301227bbb9ce7f13e46ea27 :)


Preventing diagonal moves

Is it possible to limit path steps to up, down, left and right without diagonal moves?

Hi James
The current implementation does just that.
Justin
On 20 January 2016 at 14:30, James Lovejoy<EMAIL_ADDRESS>wrote:

Is it possible to limit path steps to up, down, left and right without
diagonal moves?
—
Reply to this email directly or view it on GitHub
https://github.com/justinhj/astar-algorithm-cpp/issues/13.

Sorry yes, the bug was in my own code. Thank you for the help.


[AutoPR graphrbac/data-plane] Adding appRole property to ApplicationCreateParameters and ApplicationUpdateParameters

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/3165

(message created by the CI based on PR content)


Analyzer fails to parse use of generalized function type

At f1f884011fec6b08874951cb617a706d132c150e, the analyzer fails to parse this:
class C {
  foo(List<String Function(String)> functions) => null;
}

main() {
  new C().foo(null);
}

/cc @floitschG

Right, the informal spec of generic function type aliases (which includes function types on the form .. Function(..)) is not yet LGTM'd (see https://codereview.chromium.org/2841483003/), but all recent versions of that document have agreed that function types can be used as actual type arguments. (Generic function types are rejected as type arguments, but that's not a grammatical issue, it's a separate compile-time check). So the given example should be accepted.

@eernstg I'm a bit surprised by this. This has already been implemented in Fasta and dart2js, and there are tests committed that use this.

The informal spec has been landed in commit 5904409d50fa9b800d015faadbe49cb13e358644.

Analyzer now parses this.


Need guidance to wireless adapter configuration

Please take a look at my configurations and guide me if I have done anything wrong? I assume the hardware is working fine, my OPNSense box can see surrounding WIFI networks from "Interfaces->Wireless->Status", and my laptop can detect the surrounding WIFI networks, but my laptop can't see my OPNSense box. Here are my settings.
OPNSense UI Wireless Settings ---------------------------------------------------
(left blank or default where not mentioned)
Enable Interface yes
Static IPv4
Speed and duplex Default (no preference, typically autoselect)
IPv4 address <IP_ADDRESS>
IPv4 Upstream Gateway None
Standard 802.11g
Protection mode off
Transmit power 18
Antenna Settings Default Auto Auto
Distance Setting 100
Regulatory domain ROW
Country Australia -- (AU ROW)
Location Indoor
Mode Infrastructure (BSS)
SSID OPNTest
Enable WME yes
Enable WPA
WPA pre-shared key *********
Authentication Open System Authentication
WPA Pairwise AES Recommended
Key Rotation 60
Master Key Regen 3600
APPLY CHANGES
Hardware Factory Specs----------------------------------------------------------------
TP-Link TL-WN651G 108Mbps Super G Wireless PCI Adapter
http://www.mondoplast.net/datasheets/TL-WN651G.html
Protocol and Standards: IEEE 802.11g; IEEE 802.11b; CSMA/CA with ACK
Interface: 32-bit PCI 2.2
Frequency: 2.4~2.4835GHz
Radio Data Rate: 108/54/48/36/24/18/12/9/6/11/5.5/3/2/1Mbps (Auto Rate Sensing)
Spread Spectrum: DSSS
Modulation:1M DBPSK; 2M DQPSK; 5.5M, 11M CCK; 6M, 9M, 12M, 18M, 24M, 36M, 48M, 54M, 108M OFDM;
Transmission Distance:Indoor up to 100m, outdoor up to 300m
Transmit Power: 18dBm(Typical)
Data Security: WPA; 64/128/152 BIT WEP; TKIP/AES
Power Consumption: Typically 685mA in full Transmit (TX), 515mA in full Receive (RX)
Side Note:
Am I supposed to raise this kind of question here in Github Issues? Or do we post them in other channels, such as Stackoverflow? Where is the best place to ask questions? Thank you :)

Is the SSID public or did you check the hide option?
The forum would be better for support issues as other people will help you there as well. We use GitHub for code improvements mostly.

Thanks fichtner. I will ask future questions in the forum.
"Hide SSID" option is unchecked. As I want it to broadcast. So, other settings look okay to you?
Thanks.

Morning, can you try another standard like 802.11b?

This moved to the forum: https://forum.opnsense.org/index.php?topic=2695


Bump image to 0.25

Bumped the image crate to 0.25 which should bring performance benefits

Main performance benefits were made in decoders and encoders. It have nothing to do with image resizing.


index.html is absent

Trying out example application and getting an error:
Request: GET /room1
** (exit) an exception was raised:
** (File.Error) could not read file stats "priv/static/html/index.html": no such file or directory
(elixir) lib/file.ex:403: File.stat!/2
(plug_cowboy) lib/plug/cowboy/conn.ex:41: Plug.Cowboy.Conn.send_file/6
(plug) lib/plug/conn.ex:458: Plug.Conn.send_file/5
(example_simple) lib/plug/router.ex:259: Example.Simple.Router.dispatch/2
(example_simple) lib/simple/router.ex:1: Example.Simple.Router.plug_builder_call/2
(plug_cowboy) lib/plug/cowboy/handler.ex:12: Plug.Cowboy.Handler.init/2
(cowboy) /Users/den/proj/membrane/membrane-demo/webrtc/simple/deps/cowboy/src/cowboy_handler.erl:41: :cowboy_handler.execute/2
(cowboy) /Users/den/proj/membrane/membrane-demo/webrtc/simple/deps/cowboy/src/cowboy_stream_h.erl:296: :cowboy_stream_h.execute/3
seems like static content was not included in repo

Closed in 6f22c1542af1b2b891a17f8d0748241545c2aff8


Undefined is not a function

Uncaught TypeError: undefined is not a functionjstree.js:260 (anonymous function)jstree.js:11 instance_counterjstree.js:13 (anonymous function)
jstree.js:112 Uncaught TypeError: undefined is not a function

Sorry, I will need more steps to reproduce this - obviously all demos on the site are working, so please share your config and steps to reproduce (if needed).

I am closing this since I cannot reproduce, there is no function on line 260 and due to no further information. If you provide more info I will reopen the ticket.