Add link checker pre-commit hook.

Add a git pre-commit link checker. Unfortunately hugo doesn't have tests to check for broken links.
The current broken links on v2 branch are as follows:
$ linkrot http://localhost:1313
"https://firefox-source-docs.mozilla.org/security/nss/legacy/key_log_format/index.html": bad status: 404 Not Found
 - refs: http://localhost:1313/docs/recipes/how-to-intercept-and-decrypt-tls-traffic/, http://localhost:1313/guides/g8/, http://localhost:1313/guides/g8
"https://github.com/skylot/jadx": page missing fragments
- ids: download
 - refs: http://localhost:1313/guides/g3, http://localhost:1313/guides/g3/
"https://github.com/PiRogueToolSuite/colander": page missing fragments
- ids: production-environment
 - refs: http://localhost:1313/blog/monthly-report-n23-2023-12/
"https://pts-project.org/docs/pirogue/upgrade-a-pirogue/": bad status: 404 Not Found
 - refs: http://localhost:1313/blog/monthly-report-n7-2022-08/
"http://pirogue.local:3000": Get "http://pirogue.local:3000": dial tcp: lookup pirogue.local: no such host
 - refs: http://localhost:1313/guides/g1, http://localhost:1313/guides/g2/, http://localhost:1313/docs/pirogue/dashboard/, http://localhost:1313/guides/g1/, http://localhost:1313/guides/g2
"https://pts-project.org/docs/pirogue/configure-a-pirogue/": bad status: 404 Not Found
 - refs: http://localhost:1313/blog/monthly-report-n7-2022-08/
"https://pts-project.org/docs/pirogue/build-a-pirogue/": bad status: 404 Not Found
 - refs: http://localhost:1313/blog/monthly-report-n14-2023-03/
"https://pts-project.org/docs/pirogue/operate-a-pirogue/": bad status: 404 Not Found
 - refs: http://localhost:1313/blog/monthly-report-n7-2022-08/

Good catch! :grimacing:

Maybe we can add a pre-commit hook based on linkrot and a check in the CI interrupting the build if a broken link is detected.

@U039b I found that htmltest better than linkrot.

Stupid and simple pre-commit hook:
#!/bin/sh
exec 1>&2
if [ ! -f bin/htmltest ]; then
    echo "htmltest binary not found. Check its documentation https://github.com/wjdp/htmltest"
    exit 1
fi
rm -rf public/
npm run build
if ! ./bin/htmltest; then
    echo "Some links are broken"
    exit 1
fi
exit 0

@U039b Very cool!
I have clean up the links that was making htmltest results noisy in this PR
https://github.com/PiRogueToolSuite/piroguetoolsuite.github.io/pull/233

@U039b Do you still want to add a pre-hook since we already have a good github linkrot checks?


feat: databus allows arbitrarily many reads per index

TLDR: Up until now we were limited to only 1 read per entry of a databus column (see explanation of why below). This PR removes this limitation so that we can read from any row arbitrarily many times at the cost of adding one polynomial/commitment per databus column.
Explanation: The log derivative lookup relation involves an polynomial that contains inverses, i.e. I_i = (read_term_i*write_term_i)^{-1}. These inverses only need to be computed when the relation is "active", i.e. when the row in question either contains a databus read gate or data that is being read. At all other rows, we simply set the value of the inverse polynomial to 0. This allows a subrelation of the form:
read_term * write_term * inverses - inverse_exists
Where inverse_exists is a polynomial that takes 1 if the relation is active (or equivalently, if the inverse has been computed) and 0 otherwise. Therefore, if the inverse has been computed, we check that it is indeed equal to the inverse of read_term * write_term, otherwise, the subrelation contribution is trivially 0. If we only allow a single read from each row of a bus column, the term inverse_exists can be computed as an algebraic OR of the form:
is_read_gate + read_counts - (is_read_gate * read_counts)
since both is_read_gate and read_counts are both boolean. If read_counts is no longer boolean, no such algebraic expression exists. The solution is to introduce a dedicated boolean polynomial read_tag whose values are given by min(1, read_counts), i.e. 1 if one or more reads have been performed at that row, and 0 otherwise.
Closes https://github.com/AztecProtocol/barretenberg/issues/937

Benchmark results
Metrics with a significant change:

app_circuit_proving_time_in_ms (FPC:fee_entrypoint_public): 16,559 (+93%)
protocol_circuit_witness_generation_time_in_ms (base-parity): 340 (-55%)
protocol_circuit_proving_time_in_ms (public-kernel-app-logic): 661 (-35%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
64 txs

l1_rollup_calldata_size_in_bytes
740
740
740

l1_rollup_calldata_gas
6,644 (-1%)
6,668
6,680

l1_rollup_execution_gas
586,829
586,853
586,865

l2_block_processing_time_in_ms
1,608 (-1%)
6,051 (-1%)
12,061 (-1%)

l2_block_building_time_in_ms
35,246 (-1%)
138,203 (-1%)
275,559 (-1%)

l2_block_rollup_simulation_time_in_ms
35,029 (-1%)
137,430 (-1%)
274,043 (-1%)

l2_block_public_tx_process_time_in_ms
16,475 (-1%)
70,496
141,358 (-1%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
18,050 (+1%)
33,589

node_database_size_in_bytes
21,393,488
37,965,904

pxe_database_size_in_bytes
29,868
59,425

Circuits stats
Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
512 leaves
1024 leaves
2048 leaves
4096 leaves
32 leaves

batch_insert_into_append_only_tree_16_depth_ms
13.2
21.4 (-1%)
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.7
31.8
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.772
0.659
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
62.0 (+1%)
97.2
316 (-1%)
610
1,194 (-1%)
2,373 (-1%)
N/A

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
95.9
159
543
1,055
2,079
4,127
N/A

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.637 (+1%)
0.601
0.575 (-1%)
0.571
0.567 (-1%)
0.567 (-1%)
N/A

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
73.8
143 (-1%)
455 (-1%)
894
1,775 (-1%)
3,540 (-1%)
N/A

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
106
208
692
1,363
2,707
5,395
N/A

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.646
0.645
0.620 (-1%)
0.617
0.616 (-1%)
0.617 (-1%)
N/A

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
80.7

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
108

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.710

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
84,315
673,143

Transaction size based on fee payment method
| Metric |  |
| - |  |
Transaction processing duration by data writes.

Metric
0 new note hashes
1 new note hashes
2 new note hashes

tx_pxe_processing_time_ms
37,661 (+2%)
3,659 (-1%)
114,491 (+2%)

Metric
0 public data writes
1 public data writes
2 public data writes
4 public data writes
8 public data writes

tx_sequencer_processing_time_ms
1,310 (+1%)
2,253
1,898
2,848
1,813

The recursion tests seem to fail between commits c5ee3f0


Delete correct operator roles and support hypershift

What
Adds support for hypershift credential requests coming from subscription
Why
Hypershift roles were not being deleted and there was a chance to identify the wrong cluster when deleting operator roles

LGTM thanks

LGTM

/lgtm


Publishing to cloudapp.net not working

The instructions are clear not to use a self-signed cert
I have a custom domain (CNAME associated with my cloudapp.net domain) and a CA signed cert uploaded to the app.  If I put the domain and the cert thumbnail  into the configure_cloud.ps1 script as suggested, I get the following error when I publish:
 Recovering role... Application startup task "E:\approot\Startup.cmd" > Startup.cmd.log finished successfully. [2020-05-05T16:25:39Z] Last exit time: [2020/05/05, 16:24:54.596]. Last exit code: 0. Last role exception: System.Net.Sockets.SocketException, Details: Exception: No such host is known at System.Net.Dns.GetAddrInfo(String name) at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6) at System.Net.Dns.GetHostEntry(String hostNameOrAddress) at Sample.PolicyRecordingBot.WorkerRole.AzureConfiguration.GetInstancePublicIpAddress(String publicFqdn) in C:\Users\nigel\Source\Repos\microsoft-graph-comms-samples\Samples\V1.0Samples\LocalMediaSamples\PolicyRecordingBot\WorkerRole\AzureConfiguration.cs:line 380 at Sample.PolicyRecordingBot.WorkerRole.AzureConfiguration.Initialize() in C:\Users\nigel\Source\Repos\microsoft-graph-comms-samples\Samples\V1.0Samples\LocalMediaSamples\PolicyRecordingBot\WorkerRole\AzureConfiguration.cs:line 231 at Sample.PolicyRecordingBot.WorkerRole.WorkerRole.OnStart() in C:\Users\nigel\Source\Repos\microsoft-graph-comms-samples\Samples\V1.0Samples\LocalMediaSamples\PolicyRecordingBot\WorkerRole\WorkerRole.cs:line 92 at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum) at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.<InitializeRole>b__0() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() '.
Any suggestions as to how to get these samples working using Cloud Services (classic)?
Thanks

Sounds like the same issue as #218

Thanks, solved it.  In the end, once I'd solved the DNS issue, it was a misconfigured certificate.  Always read the error message!


wwwroot/fonts

Why fonts folder once cleared and never be created again?

Clear:
plugins: [         new CleanWebpackPlugin(             [                 './wwwroot/dist',                 './wwwroot/fonts',
New place - wwwroot/assets/:
{                 test: /\.(png|jpg|gif|ico|woff|woff2|ttf|svg|eot)$/,                 exclude: /node_modules/,                 loader: "file-loader?name=assets/[name]-[hash:6].[ext]",             },

If use FontAwesome, need add to webpack.*.js  for all ext (woff, woff2, eot, svg, ttf)
{ test: /\.woff(\?v=\d+\.\d+\.\d+)?$/, exclude: /node_modules/, loader: "file?name=assets/[name]-[hash:6].[ext]" }

@maksir I'll have a look,, could be a bug, need to check it out
Greetings Damien

fixed, the fonts folder and contents are no longer removed.


r2ai list non existent models

r2ai -mm is expected (according to doc -h) to list all downloaded models:
[r2ai:0x000061d0]> -mm
     ??? TheBloke/Mistral-7B-Instruct-v0.1-GGUF
   2.96G cognitivecomputations/dolphin-2.9.4-llama3.1-8b-gguf
     ??? segolilylabs/Lily-Cybersecurity-7B-v0.2-GGUF
     ??? FaradayDotDev/llama-3-8b-Instruct-GGUF
  11.94G ibm-granite/granite-20b-code-instruct-8k-GGUF

In reality, I only have those models:
$ ls -al ~/.local/share/r2ai/models/
total 15624388
drwxr-xr-x 2 axelle axelle        4096 Sep 17 10:07 .
drwxr-xr-x 3 axelle axelle        4096 Sep 17 10:07 ..
-rw-r--r-- 1 axelle axelle<PHONE_NUMBER> Aug 30 15:27 dolphin-2.9.4-llama3.1-8b-Q2_K.gguf
-rw-r--r-- 1 axelle axelle<PHONE_NUMBER>0 Sep 17 10:07 granite-20b-code-instruct.Q4_K_M.gguf
-rw-r--r-- 1 axelle axelle          89 Aug 30 11:40 llama.log

I think that TheBloke/Mistral-7B-Instruct-v0.1-GGUF  , segolilylabs/Lily-Cybersecurity-7B-v0.2-GGUF and  FaradayDotDev/llama-3-8b-Instruct-GGUF   were downloaded by me before, but deleted. Looks like I didn't delete them properly for r2ai?

I tried to delete a model using -m- but it didn't work:
r2ai:0x000061d0]> -m-segolilylabs/Lily-Cybersecurity-7B-v0.2-GGUF
[r2ai:0x000061d0]> -mm
     ??? TheBloke/Mistral-7B-Instruct-v0.1-GGUF
   2.96G cognitivecomputations/dolphin-2.9.4-llama3.1-8b-gguf
     ??? segolilylabs/Lily-Cybersecurity-7B-v0.2-GGUF
     ??? FaradayDotDev/llama-3-8b-Instruct-GGUF
  11.94G ibm-granite/granite-20b-code-instruct-8k-GGUF

the meaning of ??? is that the model was registered in the .r2ai.model file from your home, but the file in disk was removed. we can purge them from the json automatically. but maybe we also want to know which files are in the models directory that are not registered by the json.
and yeah you cant remove a model that's not in disk. this can be fixed but maybe we can just silently update the json and hide those ??? . what do you think?

I think I'd silently update the json to remove the ???, and only leave it to advanced end-users to edit .r2ai.model.


Automatic Mixed Precision and XLA not working with model.fit_generator

System information

**OS Platform and Distribution **: tensorflow/tensorflow:nightly-gpu-py3 Nvidia Docker image (ec33d38d1b43)
TensorFlow version (use command below): 2.1.0-dev20191106
Python version: Python 3.6.8
CUDA/cuDNN version: 10.0
GPU model and memory: Titan RTX 24GB

Describe the problem
When using model.fit_generator(), Automatic Mixed Precision (AMP) and compiling XLA doesn't seem to work. However, it all works fine with model.fit().
Please see below for a complete code sample to reproduce this.
Source code / logs
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import Sequence
import numpy as np
import random

def mixed_precision_test(use_generator):
    # use XLA
    tf.config.optimizer.set_jit(True)

    input_shape = (100, 100, 100)
    n_samples = 1000

    # build the model
    model = Sequential()
    model.add(Dense(16, input_shape=input_shape, activation='relu'))
    model.add(Flatten())
    model.add(Dense(8, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))

    optimiser = Adam(lr=0.001)

    # use AMP
    optimiser = tf.train.experimental.enable_mixed_precision_graph_rewrite(
        optimiser)

    model.compile(optimizer=optimiser,
                  loss='binary_crossentropy',
                  metrics=['accuracy'])

    if use_generator:
        class DataGen(Sequence):

            def __len__(self):
                return n_samples

            def __getitem__(self, index):
                _x = np.random.rand(1, *input_shape)
                _x = np.array(_x, dtype='uint8')
                _y = np.array([random.choice([0, 1])], dtype='uint8')
                return _x, _y

        model.fit_generator(generator=DataGen())
    else:
        x = np.random.rand(n_samples, *input_shape)
        x = np.array(x, dtype='uint8')
        y = np.array([random.choice([0, 1]) for _ in range(len(x))],
                     dtype='uint8')

        model.fit(x, y)

if __name__ == '__main__':
    mixed_precision_test(use_generator=False)
    mixed_precision_test(use_generator=True)

When setting use_generator=False, TensorFlow prints out the following logs:
2019-12-17 15:51:43.549128: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1857] Converted 26/409 nodes to float16 precision using 2 cast(s) to float16 (excluding Const and Variable casts)
2019-12-17 15:51:43.755273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-17 15:51:45.048967: I tensorflow/compiler/jit/xla_compilation_cache.cc:242] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

indicating that AMP and XLA are working as intended.
When setting use_generator=True, those logs are not present and GPU memory consumption is higher, suggesting that no casting to FP16 was performed.

According to this SO post, fit_generator is deprecated as of TF 2.1.0. It suggests passing the generator to model.fit instead. I am not able to test this right now, but I'll try to update as soon as I can.

According to this SO post, fit_generator is deprecated as of TF 2.1.0. It suggests passing the generator to model.fit instead. I am not able to test this right now, but I'll try to update as soon as I can.

For more read here https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/Model#fit_generator

Can confirm that using  model.fit(x=DataGen()) allows you to use mixed precision and XLA.


No electricity dataset

Hi! Thank you for your work.
Most datasets and scripts are clear.
But I can't find the electricity dataset you used in TGTSF.
Looking forward to your reply!

Sorry for the late reply. We will release the caption script ASAP. (in this week, hopefully...) Sorry for the incontinence.

Preparation script for elec_caption dataset is uploaded! see: elec_caption.ipynb
Please feel free to reach out if you have any problem with this.

If there is no further question, I will close this issue. Please feel free to reopen it.


Center layout v3 is blurry

Noticed when using the center layout the noty is blurry you can see it on the demo page as well.

Oh, you are right. :/
I did a research and this bug is a little shitty I guess. I didn't find a proper solution. Tweaking with zoom up&scale down etc is not good. :/
For centering calc(-50% - .5px) is the best possible solution I guess. Half pixel kinda acceptable.

fixed in v3.0.1

Still here...

without translate(calc(-50% - .5px), calc(-50% - .5px)) translateZ(0) scale(1, 1)

=> in last version (v3.2.0-beta) in last version FF on windows 10.


dipole-source

minor updates to the dipole source code so that electric dipole sources can be included and it doesn't matter if the a or b electrode is on top or bottom

Codecov Report

Merging #13 into master will decrease coverage by 0.04%.
The diff coverage is 14.28%.

@@            Coverage Diff             @@
##           master      #13      +/-   ##
==========================================
- Coverage   43.11%   43.07%   -0.05%
==========================================
  Files           9        9
  Lines        1735     1739       +4
==========================================
+ Hits          748      749       +1
- Misses        987      990       +3

Impacted Files
Coverage Δ

casingSimulations/sources.py
64.89% <14.28%> (-0.51%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 08572e7...ece57ff. Read the comment docs.


Add str method to EvalMetric to display the metric values

Can one of the admins verify this patch?

Fixed, please merge


when fuel truck kavala has been damaged, CO part of radio conversation is mute

Should be fixed. Closing.


More capacity needed in wds array to avoid all overruns.

The worst-case use of the descriptor set is larger than I first thought: 2 * nr-of-samplers + 2.
This patch adjusts that range, so no overflow of wds[] array is caused.

Cleanup your branch: This branch cannot be rebased due to conflicts


[feat]: [CDS-45294]: Added spot connector

We have added spot connector.
Describe your changes
Comment Triggers

PR Check triggers

Build: trigger build
Sub Category Field Check: trigger subcategoryfieldcheck

Please add a .txt file changelog folder correponding to your pr number describing your changes


Clustered forward rendering

Objective
Implement clustered-forward rendering.
Solution
FIXME - in the interest of keeping the merge train moving, I'm submitting this PR now before the description is ready. I want to add in some comments into the code with references for the various bits and pieces and I want to describe some of the key decisions I made here. I'll do that as soon as I can. Anyone reviewing is welcome to add review comments where you want to know more about how something or other works.

The summary of the technique is that the view frustum is divided into a grid of sub-volumes called clusters, point lights are tested against each of the clusters to see if they would affect that volume within the scene and if so, added to a list of lights affecting that cluster. Then when shading a fragment which is a point on the surface of a mesh within the scene, the point is mapped to a cluster and only the lights affecting that clusters are used in lighting calculations. This brings huge performance and scalability benefits as most of the time lights are placed so that there are not that many that overlap each other in terms of their sphere of influence, but there may be many distinct point lights visible in the scene. Doing all the lighting calculations for all visible lights in the scene for every pixel on the screen quickly becomes a performance limitation. Clustered forward rendering allows us to make an approximate list of lights that affect each pixel, indeed each surface in the scene (as it works along the view z axis too, unlike tiled/forward+).
WebGL2 is a platform we want to support and it does not support storage buffers. Uniform buffer bindings are limited to a maximum of 16384 bytes per binding. I used bit shifting and masking to pack the cluster light lists and various indices into a uniform buffer and the 16kB limit is very likely the first bottleneck in scaling the number of lights in a scene at the moment if the lights can affect many clusters due to their range or proximity to the camera (there are a lot of clusters close to the camera, which is an area for improvement). We could store the information in textures instead of uniform buffers to remove this bottleneck though I don’t know if there are performance implications to reading from textures instead if uniform buffers.
Because of the uniform buffer binding size limitations we can support a maximum of 256 lights with the current size of the PointLight struct
The z-slicing method (i.e. the mapping from view space z to a depth slice which defines the near and far planes of a cluster) is using the Doom 2016 method. I need to add comments with references to this. It’s an exponential function that simplifies well for the purposes of optimising the fragment shader. xy grid divisions are regular in screen space.
Some optimisation work was done on the allocation of lights to clusters, which involves intersection tests, and for this number of clusters and lights the system has insignificant cost using a fairly naïve algorithm. I think for more lights / finer-grained clusters we could use a BVH, but at some point it would be just much better to use compute shaders and storage buffers.
Something else to note is that it is absolutely infeasible to use plain cube map point light shadow mapping for many lights. It does not scale in terms of performance nor memory usage. There are some interesting methods I saw discussed in reference material that I will add a link to which render and update shadow maps piece-wise, but they also need compute shaders to work well. Basically for now you need to sacrifice point light shadows for all but a handful of point lights if you don’t want to kill performance. I set the limit to 10 but that’s just what we had from before where 10 was the maximum number of point lights before this PR.
I added a couple of debug visualisations behind a shader def that were useful for seeing performance impact of light distribution - I should make the debug mode configurable without modifying the shader code. One mode shows the number of lights affecting each cluster by tinting toward red for few lights or green for many lights (maxes out at 16, but not sure that’s a reasonable max). The other shows which cluster the surface at a fragment belongs to by tinting it with a randomish colour. This can help to understand deeper performance issues due to screen space tiles spanning multiple clusters in depth with divergent shader execution times.

Also, there are more things that could be done as improvements, and I will document those somewhere (I'm not sure where will be the best place... in a todo alongside the code, a GitHub issue, somewhere else?) but I think it works well enough and brings significant performance and scalability benefits that it's worth integrating already now and then iterating on.

Calculate the light’s effective range based on its intensity and physical falloff and either just use this, or take the minimum of the user-supplied range and this. This would avoid unnecessary lighting calculations for clusters that cannot be affected. This would need to take into account HDR tone mapping as in my not-fully-understanding-the-details understanding, the threshold is relative to how bright the scene is.
Improve the z-slicing to use a larger first slice.
More gracefully handle the cluster light list uniform buffer binding size limitations by prioritising which lights are included (some heuristic for most significant like closest to the camera, brightest, affecting the most pixels, …)
Switch to using a texture instead of uniform buffer
Figure out the / a better story for shadows

I will also probably add an example that demonstrates some of the issues:

What situations exhaust the space available in the uniform buffers

Light range too large making lights affect many clusters and so exhausting the space for the lists of lights that affect clusters
Light range set to be too small producing visible artifacts where clusters the light would physically affect are not affected by the light

Perhaps some performance issues

How many lights can be closely packed or affect large portions of the view before performance drops?

Yo, what do I need to do to pull this down and try it out? I've got a demo project that supports hundreds of moving lights and objects that's been sorely in need of something like this:
https://github.com/Josh015/cubism-demo-rs

Yo, what do I need to do to pull this down and try it out? I've got a demo project that supports hundreds of moving lights and objects that's been sorely in need of something like this:
https://github.com/Josh015/cubism-demo-rs

This is based on the pipelined-rendering branch of the bevy repo. So you would need to get your demo working with that by not using the prelude, using PipelinedDefaultPlugins instead of DefaultPlugins and using things from bevy_render2 / bevy_pbr2 etc instead of the old renderer. Hopefully that’s enough to get you going. If not it may be better to wait until it is merged into main. We’re weeks away from 0.6 now. :)

Some of the reference material:

The main initial reference material as it is an accessible article:  http://www.aortiz.me/2018/12/21/CG.html
“Practical Clustered Shading” which is part 2 of https://efficientshading.com/2015/01/01/real-time-many-light-management-and-shadows-with-clustered-shading/ (Part 3 shows the shadow mapping techniques I mentioned)
The z-slicing method mentioned in the aortiz article which is from Tiago Sousa’s Siggraph 2016 talk about Doom 2016: http://advances.realtimerendering.com/s2016/Siggraph2016_idTech6.pdf

I got directional and ambient lights working, but not point lights. I suspect I'm just gonna have to wait. 😞

Okay, I got it working but I am a little confused. Is the light count supposed to still be capped at a max of 10? I thought part of the point of going with clustered rendering was to support a much larger count.

Okay, all my problems getting it working were on my end. This example is intentionally overkill, but here's a scene with 256 point lights enabled:

d:

Some of the reference material:
* The main initial reference material as it is an accessible article:  http://www.aortiz.me/2018/12/21/CG.html

* “Practical Clustered Shading” which is part 2 of https://efficientshading.com/2015/01/01/real-time-many-light-management-and-shadows-with-clustered-shading/ (Part 3 shows the shadow mapping techniques I mentioned)

* The z-slicing method mentioned in the aortiz article which is from Tiago Sousa’s Siggraph 2016 talk about Doom 2016: http://advances.realtimerendering.com/s2016/Siggraph2016_idTech6.pdf

I added these as a comment before the cluster-related code in src/light.rs as I couldn't really think of a better place to do it, but we can of course move any of it anywhere.

Am I missing something, when given a window description and providing a size,  say 500x500, you get lots of artifacts and WARN bevy_pbr2::render::light: cluster offset and count out of bounds!
Can reproduce in examples/3d/cornell_box_pipelined.rs adding a WindowDescriptor, artifacts move down the screen as aspect ratio gets greater.

Am I missing something, when given a window description and providing a size, say 500x500, you get lots of artifacts and WARN bevy_pbr2::render::light: cluster offset and count out of bounds!
Can reproduce in examples/3d/cornell_box_pipelined.rs adding a WindowDescriptor, artifacts move down the screen as aspect ratio gets greater.

You're probably not missing anything, just I hadn't found a situation where that happened outside of stress tests. My guess would be that the light sources are close to the camera, and so because there are a lot of clusters close to the camera, the lights intersect many of them, and we run out of space. This is something that speaks to using a texture for the cluster light lists instead of trying to squeeze them into a 16kB uniform. However, a simpler solution that was used in one of the reference materials was to just only have one depth slice for the first 5 units along z, and then use the 'real' exponential mapping after that. I should probably just do that. I didn't because I wanted a request for N depth slices to actually produce N depth slices, rather than N - however many depth slices are in the first 5 units + 1, and I didn't find a good mathematical way of doing that. I think that was me being a bit technically pure with the implementation and the pragmatic approach is to do exactly that - make the first slice a fixed size. I'll do that as soon as I can, probably by dynamically calculating the first depth boundary >= M units along z, and using that as the far boundary of the first slice.

Sorry mate, you're so far ahead of me on this that I hate to even take up your time.  I should have been more clear, added pic to show the result, maybe I used the term artifact incorrectly.

Images

only change was adding
    App::new()
        .insert_resource(WindowDescriptor {
            width: 500.0,
            height: 500.0,
            ..Default::default()
        })

INFO bevy_render2::renderer: AdapterInfo { name: "GeForce RTX 2080 Ti", vendor: 4318, device: 7687, device_type: DiscreteGpu, backend: Vulkan }

Sorry mate, you're so far ahead of me on this that I hate to even take up your time. I should have been more clear, added pic to show the result, maybe I used the term artifact incorrectly.
Images
only change was adding
    App::new()
        .insert_resource(WindowDescriptor {
            width: 500.0,
            height: 500.0,
            ..Default::default()
        })

INFO bevy_render2::renderer: AdapterInfo { name: "GeForce RTX 2080 Ti", vendor: 4318, device: 7687, device_type: DiscreteGpu, backend: Vulkan }

Yup, I could reproduce it when adding the window size as 500x500. Thanks for finding this issue! I’ll address it as soon as I have time to.

Ok after reading your first response a few more times I get it, thanks for all you do, gl

Am I missing something, when given a window description and providing a size, say 500x500, you get lots of artifacts and WARN bevy_pbr2::render::light: cluster offset and count out of bounds!
Can reproduce in examples/3d/cornell_box_pipelined.rs adding a WindowDescriptor, artifacts move down the screen as aspect ratio gets greater.

I was wrong in my initial response. This was actually about the way the clusters are configured. There can be maximum 4096 clusters, and I am using a constant 24 depth slices, which leaves that the number of tiles in x * number of tiles in y * 24 <= 4096 to avoid the error message you saw. I just pushed a commit that fixes this. Thanks again for reporting it! :)

Alrighty I've wrapped up my final pass on this. I think its in a good spot to merge. I do want to eventually revisit putting the Clusters calculation in the "user app" instead of in the render app, as that feels like a "renderer implementation detail". But I also agree that it would involve rethinking visibility a bit. Time to move forward!

bors r+


Playbook templates for incident creation

Required items, please complete
Change(s):

See guidance below

Reason for Change(s):

See guidance below

Version Updated:

Required only for Detections/Analytic Rule templates
See guidance below

Testing Completed:

See guidance below

Checked that the validations are passing and have addressed any issues that are present:

See guidance below

Guidance <- remove section before submitting

Before submitting this PR please ensure that you have read the following sections and filled out the changes, reason for change and testing complete sections:
Thank you for your contribution to the Microsoft Sentinel Github repo.

Details of the code changes in your submitted PR.  Providing descriptions for pull requests ensures there is context to changes being made and greatly enhances the code review process.  Providing associated Issues that this resolves also easily connects the reason.

Change(s):

Updated syntax for XYZ.yaml

Reason for Change(s):

New schema used for XYZ.yaml
Resolves ISSUE #1234

Version updated:

Yes
Detections/Analytic Rule templates are required to have the version updated

The code should have been tested in a Microsoft Sentinel environment that does not have any custom parsers, functions or tables, so that you validate no incorrect syntax and execution functions properly.  If your submission requires a custom parser or function, it must be submitted with the PR.

Testing Completed:

Yes/No/Need Help

Note: If updating a detection, you must update the version field.

Before the submission has been made, please look at running the KQL and Yaml Validation Checks locally.
https://github.com/Azure/Azure-Sentinel#run-kql-validation-locally

Checked that the validations are passing and have addressed any issues that are present:

Yes/No/Need Help

Note: Let us know if you have tried fixing the validation error and need help.

References:

Guidance for Detection checks
General contribution guidance
PR validation troubleshooting

@devikamehra and @rushriva : Please review the playbooks and provide your comments. Thanks!!

Hi @lior-tamir There is a minor typo of succesfully -> successfully in both the playbooks. Can you please look into that?


Adding identifer column to Point object for quad tree investigation

Hi, not an issue so much, as something I'm struggling with adapting.
I need the ability to have a number associated with a geo point, in order to reference back to what it relates to. I've customised the point type to have length of 3 instead of 2, with the 3rd float been the ID, so the idea was, when I ran a nearest command against the quad tree of points, I'd be able to find it's owner.
The creating of the point works, and the existing tests pass. But soon as the point is added to the quad tree, it appears to lose it's 3rd value, returning X and Y as expected, but the 3rd number is always 0.
Is there something I'm missing in the quad tree section, that is removing this 3rd column. Any help or advice on the best way to approach this would be much appreciated!

hum, I was unable to duplicate this, let me know if I am doing something wrong or if there is other code I could help debug.
I changed [2]float64 -> [3]float64 then ran:
r := rand.New(rand.NewSource(42)) // to make things reproducible
qt := quadtree.New(geo.NewBound(0, 1, 0, 1))

// insert 1000 random points
for i := 0; i < 1000; i++ {
	p := geo.NewPoint(r.Float64(), r.Float64())
	p[2] = 123
	qt.Insert(p)
}

nearest := qt.Find(geo.NewPoint(0.5, 0.5))
fmt.Printf("nearest: %+v %v\n", nearest, nearest.Point()[2])

// Output:
// nearest: POINT(0.4930591659434973 0.5196585530161364) 123

If you want to do it without modifying the library you can implement the pointer interface, like:
type OtherType struct {
	Loc *geo.Point
	ID  int
}

func (ot OtherType) Point() *geo.Point {
	return ot.Loc
}

ot := &OtherType{geo.NewPoint(r.Float64(), r.Float64()), 123}
qt.Insert(ot)

nearest := qt.Find(geo.NewPoint(0.5, 0.5))
ot2 := nearest.(*OtherType)

Sadly I just realized that the Pointer interface{ Point() *geo.Point } interface makes it so you can't just embed the *geo.Point

Thanks Paul, after seeing your code work, I found the issue was with the way I was referencing it, Thanks for getting back so quickly Paul, it's a very useful package!

cool, glad it works.

In my situation I have the path and put it to the quadtree for search and comparing with reference path by bearing. For calculate bearing I need index of point in a path.
I see it is common problem to get index of point. It is possible to add index field to point in library and may be fill it up with the index from path if it comes from? Or set index by hand in some other situations.
It will be handy instead of construct custom pointers and much simpler for newbie in golang like me.


Not Supporting python 3

PyAPNs is not supporting python version 3.Getting some syntax error ^ .Could you please guide me in resolving this or any alternative ?

Fork this project and modify yourself. I did.

Thanks AhnSeognHyun,
I Forked.Could you please guide me what & where i have to modify the code?

+1

Duplicate of https://github.com/djacobs/PyAPNs/issues/85


Type : for emoji

Hello,
Can we get an option to configure/disable the shortcut : for emoji popup?
Thanks

The setting is not exposed, but it is editable if you go into the data.json file for Make.md settings. Going to reorganize the settings a bit in the next release

On Dec 16, 2022, at 7:59 AM, Bastien Perez @.***> wrote:
Hello,
Can we get an option to configure/disable the shortcut : for emoji popup?
Thanks
—
Reply to this email directly, view it on GitHub https://github.com/Make-md/makemd/issues/16, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFHKNQPILVKXE3VADEODK3WNRRTPANCNFSM6AAAAAATA5EMOU.
You are receiving this because you are subscribed to this thread.

Thanks for the info, I have modified this file and will keep checking for future versions ;).


Is inheritance possible?

I was wondering if could subclass my models. Right now compiler throws an error about
protocol 'Mappable' requirement 'from' cannot be satisfied by a non-final class ('AdDetailModel') because it uses 'Self' in a non-parameter, non-result type position

This is interesting. What does your model hierarchy look like?

Feel free to open this with more info!


Resolve conflicts

E.g. by force-push instead of a new PR.

Could this lead to problems in case a maintainer does manual changes to the changelog file etc? On the other hand, should we support manual changes at all 🤔

It merge previously and then writes the new changelog file, so it will overwrite it anyways.


Rename GCHeapSegment to ClrSegment in docs

Thank you very much for the contribution.  I'm sorry it took me so long to look at it.


Not able to publish the App from Visual Studio

[
Does this occur consistently?
Repro steps:

Version: 1.7.0
OS: win32
OS Release: 10.0.22000
Product: Visual Studio Code
Product Version: 1.67.2
Language: en

The UI and the way we deploy the azure function seem to be changed. I am able to deploy it by clicking the function app. the old Ui, functionality was easier than this but it will be efficient

Fix has been released in version 1.7.1 of the extension.


Workers: add known issue of types/node conflict with workers-types

Gives this issue greater visibility until we can fully address it.

While this is probably the easiest recommendation in lieu of actualling fixing this in worker-types, I think it should be called out that:

This recommendation only works for folks using Node.js 20.x (not 18.x/21.x etc.)
This recommendation prevents users from using new Node.js features in 20.9+
This recommendation only works for package managers that support overrides (npm, not pnpm/yarm etc.)

@MattIPv4 thanks for calling those things out. This is on our roadmap to fix soon so the hope is that this workaround is a temporary fix.

@MattIPv4 @admah What is the best way to move forward for users using yarn?

yarn supports resolutions that should work in a similar way to npm overrides:
https://classic.yarnpkg.com/lang/en/docs/selective-version-resolutions/


Way to force use debug from file

Is there any way I can force use debug from the file itself so I don't have to pass DEBUG from the command line?

If you set process.env.DEBUG before doing any require('debug') calls then it should work.
I'd say the better way to go would be export the variable in your .bashrc file:
export DEBUG="foo*, bar*"

You can use debug.enable('*'); from your code to enable debug without setting DEBUG environment variable.

Oh ya there's that too :)

Thanks!


Handle process errors

Thanks for the helpful lib! A few ideas:

If any process exits non-zero, then it'd be great if parallel would exit non-zero. This would help when using it in pipelines.
It'd be great to have an option to exit early as soon as any process has a non-zero exit code, vs. default behavior of continuing until all processes are done.

Hi @sandinmyjoints, glad you find it helpful and thanks for the suggestions. I'll bookmark this and (asap) check what GNU Parallel works in those cases. While I don't follow it 100%, I'm not trying to reinvent the wheel but rather port the good parts of it

Ok, I checked GNU parallel docs.
About your first point, I like what they do, exiting with the amount of failed jobs as code. That shouldn't be hard to implement.
Your second point would be the --halt/--halt-on-error option. I think there's no need to support that many options, a basic boolean should do, at least to get started.

All that sounds good to me! Would solve the use cases I'm thinking of.

@sandinmyjoints Implement both points, please install 1.2.0 and let me know how it goes.
Thanks!

Awesome, thank you!


list showing only 20 items.

0
down vote
favorite
I integrated autocomplete light with my django app.
In my registry i'm doing the following:
class HostAutocomplete( autocomplete_light.AutocompleteModelBase ):
search_fields = [ 'hostname' ]
choices = Host.objects.all()
In my form i have:
hosts=autocomplete_light.MultipleChoiceField('HostAutocomplete')
Autoocomplete is working fine but the list which i'm getting contains only 20 items. Database has many more entries. What could be the issue?

http://django-autocomplete-light.readthedocs.org/en/master/api.html#autocomplete_light.autocomplete.model.AutocompleteModel.limit_choices

thank you


Enexure.MicroBus.Autofac version 3.6.2

Hi, are you able to publish a new version of Enexure.MicroBus.Autofac for version 3.6.2 with the latest fixes for .net core 2.1 as well ?
Thanks

Hi @jimmystridh
Could you please provide some more detail? I'm not sure why the current packages would not work.

Are you not able to use Enexure.MicroBus.Autofac with .net core 2.1?
Can you clarify wat you mean by "publish a new version of Enexure.MicroBus.Autofac for version 3.6.2"?

Thanks!

hi @Lavinski
this was referring to the fact that Enexure.MicroBus.Autofac is only published on nuget with version 3.6.0, referring to Enexure.MicroBus version 3.6.0. So if you just refer Enexure.MicroBus.Autofac you're going to get an old version, but as I realized I could force Enexure.MicroBus to use 3.6.2 (though not sure if there are any other drawbacks with this).
Perhaps it's prudent to keep Enexure.MicroBus.Autofac version synced with Enexure.MicroBus ?

You can use the latest versions of all the packages together. Breaking changes are only present in major version bumps (e.g. 2.X.X -> 3.X.X).


AttributeError: 'Node' object has no attribute 'output_masks'

I use Keras pretrained model VGG16. The problem is that after configuring tensorflow to use the GPU I get an error that I didn't have before when using the CPU.
The error is the following one:
Traceback (most recent call last):
File "/home/guillaume/Documents/Allianz/ConstatOrNotConstatv3/train_network.py",      line 109, in <module>
 model = LeNet.build(width=100, height=100, depth=3, classes=5)
File "/home/guillaume/Documents/Allianz/ConstatOrNotConstatv3/lenet.py", line 39,    in build
 output = model(pretrainedOutput)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py", line 443, in __call__
 previous_mask = _collect_previous_mask(inputs)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py", line 1311, in _collect_previous_mask
mask = node.output_masks[tensor_index]
AttributeError: 'Node' object has no attribute 'output_masks'

I get it after executing this code :
pretrained_model = VGG16(
    include_top=False,
    input_shape=(height, width, depth),
    weights='imagenet'
)
for layer in pretrained_model.layers:
    layer.trainable = False

model = Sequential()
# first (and only) set of FC => RELU layers
model.add(Flatten())
model.add(Dense(200, activation='relu'))
model.add(Dropout(0.5))
model.add(BatchNormalization())
model.add(Dense(400, activation='relu'))
model.add(Dropout(0.5))
model.add(BatchNormalization())

# softmax classifier
model.add(Dense(classes,activation='softmax'))

pretrainedInput = pretrained_model.input
pretrainedOutput = pretrained_model.output
output = model(pretrainedOutput)
model = Model(pretrainedInput, output)

I've got keras (2.2.2) and tensorflow(1.10.0rc1). I've also tried on keras 2.2.0 and same error. The thing is that the python environment I use works on others non-pretrained NN.
Moreover, I'm able to connect two homemade models. It's only whith the pretrained ones there is a problem and not only VGG16.

Hi Please see the libraries you imported .. import everything of keras ..do not mix up tensorflow

+1. Native keras doesn't go well with tf.keras.

I have the same problem.  I guess the output tensor of the vgg16 model cannot be used properly.
I used the output tensor of the vgg16 model in the following simple code.
The versions of Keras and TensorFlow were 2.2.2 and 1.10.0rc1, respectively.
The error messages were:
Traceback (most recent call last):
  File "test_vgg16_1.py", line 31, in <module>
    model = Model( inputs=vgg16_conv_model.input , outputs=fc_model(vgg16_conv_model.output) )
  File "/usr/local/lib64/python3.6/site-packages/keras/engine/base_layer.py", line 443, in __call__
    previous_mask = _collect_previous_mask(inputs)
  File "/usr/local/lib64/python3.6/site-packages/keras/engine/base_layer.py", line 1311, in _collect_previous_mask
    mask = node.output_masks[tensor_index]
AttributeError: 'Node' object has no attribute 'output_masks'

I ran the following code:
#
# Setting the image attributes
#
resized_image_height = 150
resized_image_width = 150
no_image_channels = 3
max_pixel_value = 255

#
# Setting the convolutional network of the VGG16 model
#
from keras.applications.vgg16 import VGG16
vgg16_conv_model = VGG16( include_top=False , # remove the fully-connnected network
                          weights='imagenet' ,
                          input_shape=( resized_image_height , resized_image_width , no_image_channels ) )

#
# Setting the new fully-connected network
#
from keras.models import Sequential, Model
from keras.layers import Flatten, Dropout, Dense
fc_model = Sequential()
fc_model.add(Flatten(input_shape=vgg16_conv_model.output_shape[1:]))
fc_model.add(Dropout(0.2))
fc_model.add(Dense( 512 , activation='relu' ))
fc_model.add(Dense( 1 , activation='sigmoid' ))

#
# Combining the models
#
model = Model( inputs=vgg16_conv_model.input , outputs=fc_model(vgg16_conv_model.output) )

I would appreciate it if you could give me some advice to resolve this error.

Nightly TF builds are, hmm, nightly, so Keras haven't become compatible with all its features yet.
If using latest version of TF isn't important for you, downgrade to TensorFlow 1.9, there's no this bug.

@arquolo you are right! I have the same problem. I downgrade to TensorFlow 1.9 with Keras 2.2.2, it's OK!

I have TF 1.9 and keras 2.2.2 and it's now working
C:\Users\abdo_\Anaconda3\python.exe C:/Users/abdo_/PycharmProjects/TGSChallenge/model_script.py
C:\Users\abdo_\Anaconda3\lib\site-packages\h5py_init_.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from .conv import register_converters as register_converters
Using TensorFlow backend.
Traceback (most recent call last):
File "C:/Users/abdo/PycharmProjects/TGSChallenge/model_script.py", line 91, in
resizeInput=Lambda(function=resize)(imageInput)
File "C:\Users\abdo\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 731, in call
inputs, outputs, args, kwargs)
File "C:\Users\abdo_\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 848, in set_connectivity_metadata
input_tensors=inputs, output_tensors=outputs, arguments=kwargs)
File "C:\Users\abdo_\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 996, in add_inbound_node
arguments=arguments)
File "C:\Users\abdo\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 1665, in init
layer.outbound_nodes.append(self)
AttributeError: 'InputLayer' object has no attribute 'outbound_nodes'

I had a similar problem. I was trying to use VGG16 and got this error AttributeError: 'InputLayer' object has no attribute 'outbound_nodes' .
My tensor flow-gpu version was 1.11 and after I downgraded it to 1.9  according to @arquolo it works now! Thanks a lot!

I have encountered the same problem. The most minimal code example already reproduces it:
from keras.layers import GlobalAveragePooling2D
from keras.applications.inception_v3 import InceptionV3
base_model = InceptionV3(weights=None, include_top=False)
x = base_model.output
x = GlobalAveragePooling2D()(x)

The versions are:
Keras               2.2.4
Keras-Applications  1.0.4
Keras-Preprocessing 1.0.2
tensorboard         1.12.0
tensorflow-gpu      1.12.0
Is there no solution save downgrading to older versions of keras and tf-gpu?
It feels so defeatist to have to do that...

@arquolo Thanks, that did the trick!!
P.S.
I had been under the impression that the main keras package upgrades the others during piping. Obviously I could not have been more wrong.

Closing this issue since its resolved. Feel to reopen if the issue still persists. Thanks!

same issue as @abdelrahmanmohamed i am getting even after installing tensorflow-gpu 1.9.0 and keras latest version 2.2.4 as well as 2.2.2, in both the cases i am getting error saying
AttributeError: 'InputLayer' object has no attribute 'outbound_nodes'
Please help

According to this post, it appears that keras-contrib requires Keras 2.1.1

I had a similar issue, but with different architecture. As people suggested, it's important not to mix keras with tensorflow.keras, so try swapping
from keras.preprocessing import image
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D
from keras import backend as K

to:
from tensorflow.keras.preprocessing import image
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
from tensorflow.keras import backend as K

Also make sure, you don't use keras.something inside your code (not only imports) as well, hope it helps : )
Also, I used Keras 2.2.4 with tensorflow 1.10.0

keras | 2.2.5 | 2.2.4
keras-applications | 1.0.8 | 1.0.8
keras-preprocessing | 1.1.0 | 1.1.0

scipy
1.3.1
1.3.1

setuptools
39.1.0
41.0.1

six
1.12.0
1.12.0

tensorboard
1.9.0
1.14.0

tensorflow
1.9.0
1.14.0

tensorflow-estimator
1.14.0
1.14.0

tensorflow-gpu
1.9.0
1.14.0

termcolor
1.1.0
1.1.0

theano
1.0.2
1.0.3

I have the same issue ! although I downgraded everything to be compatible with the above mentioned versions !!

keras | 2.2.5 | 2.2.4
keras-applications | 1.0.8 | 1.0.8
keras-preprocessing | 1.1.0 | 1.1.0
scipy | 1.3.1 | 1.3.1
setuptools | 39.1.0 | 41.0.1
six | 1.12.0 | 1.12.0
tensorboard | 1.9.0 | 1.14.0
tensorflow | 1.9.0 | 1.14.0
tensorflow-estimator | 1.14.0 | 1.14.0
tensorflow-gpu | 1.9.0 | 1.14.0
termcolor | 1.1.0 | 1.1.0
theano | 1.0.2 | 1.0.3
I have the same issue ! although I downgraded everything to be compatible with the above mentioned versions !!

Try using either keras or tensorflow.keras, but don't mix the two of them together. This was what fixed my issue.

extremely annoying issue, I have installed the latest versions of TF and keras. But, have the same problem.
tf - '2.2.0'
keras - 2.3.0-tf

If you are using YOLO. This what you have to do. As Raseth said.
from keras.preprocessing import image
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D
from keras import backend as K
to:
from tensorflow.keras.preprocessing import image
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
from tensorflow.keras import backend as K
My only solution was to fork the GIT and edit it there.
I did not need to downgrade Keras or Tensorflow.

@luisvivasg  solution works. I used tensorflow.keras instead of keras.
I am using tensorflow-gpu=2.0.0

I have same problem.
  File "/data2/wangxinsheng/yft/yft_semantic_train_model/keras_textclassification/base/embedding.py", line 283, in build     self.output = NonMaskingLayer()(encoder_layer)   File "/data/anaconda3/envs/miss_torch_develop/lib/python3.7/site-packages/keras/engine/base_layer.py", line 443, in __call__     previous_mask = _collect_previous_mask(inputs)   File "/data/anaconda3/envs/miss_torch_develop/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1311, in _collect_previous_mask     mask = node.output_masks[tensor_index] AttributeError: 'Node' object has no attribute 'output_masks'

K!
I can't get tensorflow 1.9.
There is no tensorflow 1.9 in https://pypi.org/project/tensorflow-gpu/1.15.2/#history.

Do you know how to fix it?


Question: how to use Postfix hash: lookup table values?

Hi,
I'm trying to setup an email relay which uses the following Postfix 'hash' lookup tables (*):
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_generic_maps = hash:/etc/postfix/generic

hash: values can be set using POSTFIX_ Docker environment variables, but as the /etc/postfix directory is not exposed, I'm not sure how I can call postmap to generate the Berkeley DB files which hash: expects.
I tried to replace hash:<BerkeleyDBfile> with equivalent inline:{key=value, key=value, key=value} but without success :confused:
The inline: lookup seems supported as reported by
docker exec <container_id> postconf -m
btree
cidr
environ
fail
hash
inline
internal
memcache
nis
pipemap
proxy
randmap
regexp
socketmap
static
tcp
texthash
unionmap
unix

I guess the easiest would be to expose the /etc/postfix directory as a Docker volume?
(*) For reference, I'm following the guide at https://www.linode.com/docs/guides/postfix-smtp-debian7/
Thanks a lot,
Pete

Hi, can you give some more details like error messages or what is not working? sorry i'm not really a postfix expert
I've tried this and it seems to work:
docker run --rm -e 'POSTFIX_alias_maps=inline:{a=1,b=2,c=3}'  mwader/postfix-relay

Then check if it got configured inside the container:
root@261ea78f933b:/# postconf | grep -E ^alias_maps
alias_maps = inline:{a=1,b=2,c=3}

To expose (volume mount) files in /etc/postfix you probably have to volume mount exact path and not the whole /etc/postfix directory. Or you just put the files somewhere else.

Hi,
Thanks a lot for your quick reply.
When trying to reproduce and document the problem, I found that inline: was actually working fine, and that my configuration values were wrong. They would have been correct for an instance of Postfix running on the host, but not when running inside the container.
I corrected the values and now the container is relaying mails to the external SMTP server as expected. Great!
Just a quick question about your note

To expose (volume mount) files in /etc/postfix you probably have to volume mount exact path and not the whole /etc/postfix directory. Or you just put the files somewhere else.

I not sure I understand this. Could you please elaborate?
Thanks again,
Pete

Sure, if you would do -v /path/on/host:/etc/postfix i'm not sure what would happen as there are other postfix config files in that directory and also the container will write to it during startup to convert POSTFIX_* variables to configuration. So if you want to have hash files somewhere i would recommend to either put them somewhere else or use full file path volume mount -v /path/to/hash/file:/etc/postfix/hashfile.

Did you get it working? Close issue?

Sorry for my late answer. Your explanation is clear, thank you. Also, thank you for the work you put in this image. A real timesaver! :)
Actually, as the inline: lookups work as expected now, I did not investigate the /etc route any further, so we can close this issue.
Thanks again,
Pete

No problem, good to hear! 👍


Compat update for String rename

This one look ok other than the missing Compat version bump.

Coverage remained the same at 35.2% when pulling 79ba136839227680bc2367e3ce1c9243a3778271 on StefanKarpinski:master into 0cf26031a5b1c270a91507691bcab7ca4c7429f4 on JuliaWeb:master.

Coverage remained the same at 35.2% when pulling 00feba677e3dbd40e69573f8ee5372e97603d4bc on StefanKarpinski:master into 0cf26031a5b1c270a91507691bcab7ca4c7429f4 on JuliaWeb:master.

Bump

@malmaud Can this one be merged?


[CMS-390] Add support for empty upstream-based sites

I'm guessing empty support for conversion:restore-master will be handled in a separate ticket/PR, right? I think it should be mostly switching back to the right upstream but you know better than me

I'm guessing empty support for conversion:restore-master will be handled in a separate ticket/PR, right? I think it should be mostly switching back to the right upstream but you know better than me
Actually, conversion:restore-master does include an upstream switching. Going to address this.Thanks for rising this!

I'm guessing empty support for conversion:restore-master will be handled in a separate ticket/PR, right? I think it should be mostly switching back to the right upstream but you know better than me

Actually, conversion:restore-master does include an upstream switching step. Going to address this. Thanks for raising this!

Updated the PR to include this stuff


[WIP] sn0int: init at 0.18.2

Motivation for this change
sn0int is  a semi-automatic OSINT framework and package manager

https://github.com/kpcyrd/sn0int

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg bulid sn0int

@GrahamcOfBorg build sn0int

@GrahamcOfBorg build sn0int

@GrahamcOfBorg build sn0int

@GrahamcOfBorg build sn0int

@GrahamcOfBorg build sn0int

@GrahamcOfBorg build sn0int

@GrahamcOfBorg build sn0int

Hi, @worldofpeace I think this PR got stuck, too. Could you review this PR?
Thank you very much!

Hi @worldofpeace, thanks for the review.
Is there any improvement I can do for this PR?

Hi @worldofpeace, thanks for the review.
Is there any improvement I can do for this PR?

Oops, thanks for reminding me. Could you add what you mentioned above into a comment (doCheck)?

Hi @worldofpeace, thanks for the review!
The commit has been updated as your suggestion.

@xrelkd Merged :sparkles:

@worldofpeace Thank you!!
sn0int has been added into nixpkgs for #81418 (cc @JoshuaFern)


Status code for ACL access restrcitction

I guess we can have one or two (for basic and extended ACL) status codes that notify about access restriction. It can be used in https://github.com/nspcc-dev/neofs-s3-gw/pull/328 to return correct status code (now it relies on message with access to operation * is denied by sub string).

It would be nice to have status code for not found container and object errors as well.
I'm not sure if we need separate issue for this.


Encrypting a model in Mediapipe

Hi, Apple recently enabled the encryption of CoreML models during app development.
https://developer.apple.com/documentation/coreml/core_ml_api/encrypting_a_model_in_your_app
We are currently inching toward deployment and protecting model IP is rather important to us.  Hopefully this can be done within Mediapipe!   While I haven't spent a lot of time on the internals, I believe within Mediapipe the model is packaged as a TFlite file rather than .mlmodelc, and the pipeline interfaces with coreml through the delegate, so I am not sure how to access Apple encryption.
Can we encrypt models using the mediapipe platform?  If so how? Is there something similar for Android as well?
regards,
Martin
(note: I was encouraged to repost this from the group forum here. I will link if one gets answered!)

Model encryption is currently not on MediaPipe roadmap. However, you could consider using a 3rd-party encryptor (e.g., see answers in https://github.com/tensorflow/tensorflow/issues/21501) to encrypt your TFLite model for distribution. You can then, for example, modify TfLiteModelCalculator to incorporate the associated decryption library to restore the TFLite model file, before sending it out as a MODEL side packet to be consumed by TfLiteInferenceCalculator.

Ok thanks for the swift reply.
Encryption is well outside my realm of expertise but I suspect the suggestion would be considerably more vulnerable to cracking than using the OS encryption.
Need to contemplate how that impacts our pipeline, and whether writing a CoreMLInferenceCalculator for use on IOS might be easier!
Anyhow I will close the ticket. Thanks again for swiftness.


[SOL logging]  SOL log script

spawn ssh -p 2200<EMAIL_ADDRESS>@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
b1:b5:45:c1:28:cd:02:12:bf:31:91:7f:84:2d:9c:ba.
Please contact your system administrator.
Add correct host key in /gsa/ausgsa/home/h/o/hostboot/.ssh/known_hosts to get rid of this message.
Offending RSA key in /gsa/ausgsa/home/h/o/hostboot/.ssh/known_hosts:133
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
X11 forwarding is disabled to avoid man-in-the-middle attacks.
Permission denied (publickey,password).
**ERROR** End of data waiting for password prompt.

Closing this as there are mechanism to by pass this..


Read before filing is out of date.

The "READ BEFORE FILING" information that is provided when attempting to submit an issue is out of date.  (Kind of meta, I know)
It states that to log "Everything else. https://connect.microsoft.com/VisualStudio/feedback/LoadSubmitFeedbackForm", however Microsoft Connect has been retired, aka shutdown.
In addition  "https://github.com/dotnet/corefx" is also out of date, as that link says "Going forward, the .NET team is using https://github.com/dotnet/runtime to develop the code and issues formerly in this repository."
Please update this section to the correct information.
Thank you.
READ BEFORE FILING
Do not file issues regarding functionality of the .NET Framework. This repo
only tracks infrastructure issues regarding the following items:

Missing or outdated source in this repository
Reference Source Browser (http://referencesource.microsoft.com/)
Debugging the .NET Framework using Reference Source

For issues regarding functionality, please use the following resources:

BCL. https://github.com/dotnet/corefx
ASP.NET. https://github.com/aspnet/home
Everything else. https://connect.microsoft.com/VisualStudio/feedback/LoadSubmitFeedbackForm

I am surprised that this GitHub setup has not been corrected yet.


feat: check pipelines' tokens before executing/creating pipeline

⚠ Pre Checklist

Please complete ALL items in this checklist, and remove before submitting

[x] I have read through the Contributing Documentation.
[ ] I have added relevant tests.
[ ] I have added relevant documentation.
[x] I will add labels to the PR, such as pr-type/bug-fix, pr-type/feature-development, etc.

Summary
What does this PR do?

Check tokens before creating pipelines
Restranform will succeed regardless of whether the tokens are  legal or not

Does this close any open issues?
Related #8017
Screenshots
Include any relevant screenshots here.
Github
Collect data with wrong token

restransform can be executed successfully

Collect Data In Full Refresh Mode

Other Information
Any other information that is important to this PR.

There still are some problems with GitHub connection, I am working on it.


Frontend web development html css js: Domain name basics

Description 📜
This PR explains basics of Domain name with examples
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
Fixes #3208

Type of change 📝

[ ] Au#dio (Should be in mp3 format Includes speech clarity, Concise ,Low distortion)
[ ] Vi#deo (Animations, screen-recordings, presentations and regular explanatory films are all possibilties etc)
[x] Documentation (Content Creation in the form of codes or tutorials)
[ ] Other (If you choose other, Please mention changes below)

Domain of Contribution 📊

[ ] Android Dev #(Flutter)
[ ] Android Dev #(Java)
[ ] Android Dev #(Kotlin)
[ ] Backend Dev #(Java)
[ ] Backend Dev #(.NET)
[ ] Backend Dev #(PHP)
[ ] Backend Dev #(Python)
[ ] C/#CPP
[ ] Competitive #Programming
[ ] Cyber #Security
[ ] DS#A
[ ] Data#base
[ ] Datascience with #Python
[ ] Datascience with #R
[x] Frontend Dev HTML/CSS/JS
[ ] Frontend Dev #React/Angular/Vue
[ ] Go#lang
[ ] Interview #Prep
[ ] Java_#Domain
[ ] Java#Script
[ ] ME#RN
[ ] Machine #Learning
[ ] Open #Source
[ ] Python_#Domain
[ ] Ru#st
[ ] Statis#tics
[ ] UI/#UX

Checklist ✅

[x] I follow Contributing Guidelines & Code of conduct of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I'm GWOC'21 contributor

Screenshots / Gif (Optional) 📸

Great! Let's wait for the supervisor's approval!

Is this issue still live? I'd love to contribute to this!

Hey Kaushal! Please check the issues tab for more issues. You cannot contribute to the issue linked with this PR.


Use official Docker tag for dind

As in https://github.com/codeship/codeship-tool-examples/pull/18

Thanks @fabian!


action bar splitting and number of action buttons should be determined only by the device independent resolution

In beta 5 you disable splitting the action bar on large screens, but when a large 1024x600 screen uses HDPI it is only 400dip wide in portrait mode.  That is much too small to fit a logo, title, four action buttons, and the overflow button.

I'll have to write a wrapper of some sort around getResources().getBoolean() for pre-4.0 platforms that provides hard coded values for checks against the display metrics to determine the smallest-width dp and regular dp resource qualifiers. I initially had something like this but had hoped I could get away with things like the 'normal' and 'large' metrics. Oh well.


Countdown option for recording

Create option to countdown 5 seconds before beginning recording

Reference: http://itechmasters.blogspot.com/2011/11/how-to-make-programmed-self-timer.html


DAVE-301 Refactor Hebung Frameworks Java

name: Pull request template
about: Create a pull request to contribute to our software
title: ""
labels: ""
assignees: ""
Pull Request
Description
Reference
Issues:
Definition of Done

[ ] Acceptance criteria are met
[ ] Build/Test workflow has successfully finished
[ ] Release notes are complemented
[ ] Documentation is complemented (operator manual, system specification)
[ ] Frontend is locally smoke-tested
[ ] Side branches are deleted
[ ] Board is updated
[ ] Infrastructure is adjusted

Elasticsearch anpassen:
https://docs.spring.io/spring-data/elasticsearch/reference/elasticsearch/clients.html

Der aufgedröselte Code für die Elasticsearch-API 7.X.X.

        final String prefix = new String(queryBuilder);

        final int maxHits = 3;
        final String zaehlstelle_suggest = "zaehlstelle-suggest";
        final CompletionSuggestionBuilder suggest = SuggestBuilders
                // Field name: suggest
                .completionSuggestion("suggest")
                // Der Prefix für die completion suggestion
                .prefix(query, Fuzziness.ZERO)
                .skipDuplicates(true)
                .size(maxHits);
        final SearchResponse searchResponse = this.elasticsearchOperations.suggest(
                // die completion suggestion query
                new SuggestBuilder().addSuggestion(
                        // every suggestion needs a name: zaehlstelle-suggest
                        zaehlstelle_suggest,
                        suggest
                ),
                // ermittelt den Index der verwendet werden soll -> Indexname: suggestion
                this.elasticsearchOperations.getIndexCoordinatesFor(CustomSuggest.class)
        );
        final List<SucheWordSuggestDTO> result = new ArrayList<>();

        // Extrahieren der gefundenen suggestions
        final List<? extends Suggest.Suggestion.Entry.Option> options = searchResponse
                .getSuggest()
                .getSuggestion(zaehlstelle_suggest)
                .getEntries()
                .get(0)
                .getOptions();


Project not available on jitpack.io

Please fix it. I soooooo need this library!

I am not having a problem with it, can you paste your gradle file please.

Had the same issue, this from your ReadMe "compile 'com.github.Kennyc1012:BottomSheet:-2.1.2-g26cb036-1'" is not working, while this is working "compile 'com.github.Kennyc1012:BottomSheet:2.1.2"


no error for out of block-scope variable in debug mode

Steps to reproduce
Run this shell command:
vim -Nu NONE -S <(tee <<'EOF'
    vim9script
    def Func()
        if true
            var n = 123
        endif
        eval 0 + 0
    enddef
    breakadd func 4 Func
    Func()
EOF
)

At the debug prompt, execute echo n:  123 is echo'ed.
Expected behavior
An error is given:
E121: Undefined variable: n

Nothing is echo'ed.
Because n is set in the scope of the previous if block.  And at the time we ask for its value, we're outside the block, hence out of scope.
Version of Vim
9.0 Included patches: 1-499
Environment
Operating system: Ubuntu 20.04.5 LTS
Terminal: XTerm(353)
Value of $TERM: xterm-256color
Shell: GNU bash, version 5.0.17
Additional context
N/A

Steps to reproduce
Run this shell command:
vim -Nu NONE -S <(tee <<'EOF'
    vim9script
    def Func()
        if true
            var n = 123
        endif
        eval 0 + 0
    enddef
    breakadd func 4 Func
    Func()
EOF
)

At the debug prompt, execute echo n:  123 is echo'ed.
Expected behavior
An error is given:
E121: Undefined variable: n

Nothing is echo'ed.
Because n is set in the scope of the previous if block.  And at
the time we ask for its value, we're outside the block, hence out of
scope.

Strictly speaking the value of "n" is not available here.  But it does
exist, so why not show the last value?
There is a mechanism to avoid showing variables that haven't been
declared yet, thus if you would try to show "n" before line 2 it won't
work.  Adding a mechanism to hide variables after their scope has been
left is quite complicated.  And I don't think it is all that useful.
--
hundred-and-one symptoms of being an internet addict:
97.
/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

And I don't think it is all that useful.

Understood, closing then.
Something which would be more useful is for the line range to be passed to the debug command so that a multiline statement including a comment is not truncated too early.  It was already discussed here.


Update README.md to version 0.7.0 apis

Fix api change from ParquetFileWriter constructor to ParquetWriter.openFile static method
Fix api changes from .end() to .close() method name
Fix new TIMESTAMP constant name
Add await expression for all async functions

Many thanks!


Annotations missing in devel

in devel the comments are gone (which inputs and outputs are used for each invoke call), also it is no longer visible which operation is called (fix by defining and using an enum into g_registration)

Wouldn't these changes conflict with the vision of having loops instead of linear code?

Good point. I will probably need to put them into a different place (e.g. annotations on the arrays instead of on the code (which will probably be gone soon))

Related to #13

After implementing the reworked code I can only envision a comment with pseudo-code describing the network in human readable form:
E.g.
/*    Inputs  10 (1,1)
0:  1 = QUANTIZE(10)
1:  4 = FULLY_CONNECTED(1,2,3)
2:  7 = FULLY_CONNECTED(4,5,6)
3:  0 = FULLY_CONNECTED(7,8,9)
4:  11 = DEQUANTIZE(0)
Outputs  11 (1.1)  */
Do you think that this adds value to the generated file?

Since the comment would be generated, I don't see any downside. It can only help with debugging!


Java client?

Is there an official Java client for SignalR Core?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3cf283c9-5cd2-a9ff-b8c8-372d8a04356d
Version Independent ID: 729fddcf-24e4-b563-4952-a8045b231868
Content: ASP.NET Core SignalR JavaScript client
Content Source: aspnetcore/signalr/javascript-client.md
Product: aspnet-core
GitHub Login: @rachelappel
Microsoft Alias: rachelap

@mot256 At this time, no. The Java client will ship with ASP.NET Core 2.2. See https://github.com/aspnet/Announcements/issues/307.


Issue running flash

I am getting this error:
*** Parameter error: 'NoneType' object has no attribute 'bInterfaceNumber' ***
I am just running:
python3 pystlink.py flash:erase flash:verify:0x08000000:/home/automation/Downloads/ss2_firmware.bin
Note: I tried running via sudo and still same issue. I am running on Ubuntu 18.04.

Upgraded to Python 3.8 solved it


The deterministic function doesn't return proxy's address

When using the deterministic function to discover the address of an upgradable contract, it's returning the implementation address only.
See more: https://github.com/wighawag/hardhat-deploy/issues/129#issuecomment-929381609
Maybe would be useful to return both? For instance const { address, implementation, deploy } = deterministic(...) where address and implementation are the same when isn't using proxy.

fixed in 0.9.3 but let me know if hit any issues
Thanks for the report!


Add FTPFile tests and fix javadoc typos

Fixed some typos in Javadoc and adjusted documentation to code
Replaced HTML snippets with Javadoc counterparts
Moved serialization-related methods under the "serialization comment"
Adds FTPFile tests

FYI @kinow @garydgregory

Hi @jkbkupczyk
Please don't move methods in PRs, it makes your PR noisier, and takes longer to review. The methods were sorted in AB order, let's keep it that way, nice and simple, easier to find code, especially in PRs ;-)

@garydgregory done
Thought it looked kinda odd to have the read method before setters. But it makes sense to have them sorted :)


API route for /spawn

This PR adds an an API route for spawn (POST to /api/spawn) so that Thebe can use that endpoint to spawn containers in a way that works across browsers.
In order for this to work, I had to disable xsrf.
I also added the CORS setting to the /stats endpoint as well, which really should have been in that last PR.

Thanks!
I'm happy with bringing in the APISpawnHandler for programmatic usage, but we have to keep the SpawnHandler redirect for currently running sites.

Oh, yeah, that was a mistake, I meant to revert those.

OK, this removes those changes to SpawnHandler. Should be good to go now.

:shipit:
Thanks!

@zischwartz does this PR obviate the need to use https://github.com/zischwartz/tmpnb alongside thebe?
Thanks for your work on this!

Yup, that's how the thebe on https://lambdaops.com/share-cell/ works. It's using jupyter/tmpnb for the backend.

Thanks @rgbkrk!


Bun installation fails on debian

Describe the bug
Running rtx install bun on debian docker container fails to
rtx unzip ~/.local/share/rtx/downloads/bun/1.0.17/bun-linux-x64.zip -d ~/.local/share/rtx/downloads/bun/1.0.17
rtx No such file or directory (os error 2)
rtx Run with RTX_DEBUG=1 for more information

To Reproduce
Run docker build for
FROM debian:12

RUN apt-get update && apt-get install -y wget gnupg
RUN install -dm 755 /etc/apt/keyrings
RUN wget -qO - https://rtx.jdx.dev/gpg-key.pub | gpg --dearmor | tee /etc/apt/keyrings/rtx-archive-keyring.gpg 1> /dev/null
RUN echo "deb [signed-by=/etc/apt/keyrings/rtx-archive-keyring.gpg arch=amd64] https://rtx.jdx.dev/deb stable main" | tee /etc/apt/sources.list.d/rtx.list
RUN apt-get update && apt-get install -y rtx

RUN rtx install bun

Expected behavior
Bun is installed successfully.
rtx doctor output
rtx version:
  2023.12.26 linux-x64 (08cb30d 2023-12-14)

build:
  Target: x86_64-unknown-linux-gnu
  Features: DEFAULT, NATIVE_TLS, OPENSSL
  Built: Thu, 14 Dec 2023 02:47:48 +0000
  Rust Version: rustc 1.74.1 (a28077b28 2023-12-04)
  Profile: release

shell:
  bash
  GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)
  Copyright (C) 2022 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

  This is free software; you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

rtx data directory:
  /root/.local/share/rtx

rtx environment variables:
  RTX_SHELL=bash

settings:
  {"always_keep_download": "false", "always_keep_install": "false", "asdf_compat": "false", "disable_default_shorthands": "false", "disable_tools": "[]", "experimental": "false", "jobs": "4", "legacy_version_file": "true", "legacy_version_file_disable_tools": "[]", "plugin_autoupdate_last_check_duration": "7d", "raw": "false", "trusted_config_paths": "[]", "verbose": "false", "yes": "false"}

config files:

plugins:
  bun      (core)
  deno     (core)
  go       (core)
  java     (core)
  node     (core)
  python   (core)
  ruby     (core)

toolset:

No problems found

Additional context
RTX debug output:
root@33b44b7e459d:/# RTX_DEBUG=1 rtx install bun
[DEBUG] Files:
Installed Plugins:
[DEBUG] /usr/bin/rtx install bun
[DEBUG]
rtx<EMAIL_ADDRESS>downloading https://github.com/oven-sh/bun/releases/download/bun-v1.0.17/bun-linux-x64.zip                                                                                ⠁ 0s [DEBUG] GET Downloading https://github.com/oven-sh/bun/releases/download/bun-v1.0.17/bun-linux-x64.zip to ~/.local/share/rtx/downloads/bun/1.0.17/bun-linux-x64.zip
[DEBUG] GET https://github.com/oven-sh/bun/releases/download/bun-v1.0.17/bun-linux-x64.zip
[DEBUG] starting new connection: https://github.com/
rtx<EMAIL_ADDRESS>downloading https://github.com/oven-sh/bun/releases/download/bun-v1.0.17/bun-linux-x64.zip                                                                                ⠉ 0s [DEBUG] redirecting 'https://github.com/oven-sh/bun/releases/download/bun-v1.0.17/bun-linux-x64.zip' to 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/357728969/c690b109-d443-4264-941d-7cdc1fd3472c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20231214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231214T073728Z&X-Amz-Expires=300&X-Amz-Signature=3797ef46eb5075346f0871d9930795b0ae6d4a692c1db04bf1f8d2b86197b395&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=357728969&response-content-disposition=attachment%3B%20filename%3Dbun-linux-x64.zip&response-content-type=application%2Foctet-stream'
[DEBUG] starting new connection: https://objects.githubusercontent.com/
[DEBUG] GET https://github.com/oven-sh/bun/releases/download/bun-v1.0.17/bun-linux-x64.zip 200 OK
rtx<EMAIL_ADDRESS>installing /root/.local/share/rtx/downloads/bun/1.0.17/bun-linux-x64.zip                                                                                ⠒ 1s [DEBUG] $ unzip /root/.local/share/rtx/downloads/bun/1.0.17/bun-linux-x64.zip -d /root/.local/share/rtx/downloads/bun/1.0.17
Error:
   0: unzip ~/.local/share/rtx/downloads/bun/1.0.17/bun-linux-x64.zip -d ~/.local/share/rtx/downloads/bun/1.0.17
   1: No such file or directory (os error 2)

Location:
   src/file.rs:282

Version:
   2023.12.26 linux-x64 (08cb30d 2023-12-14)

Suggestion: Run with RTX_DEBUG=1 for more information.

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

That was fast! ❤
Great work with rtx ! This is awesome tool!


algoliasearch-client-javascript provide cjs, es, but does not support deduping

Hello,
We had a memory leak while building our application, and we were able to solve it by increasing the memory for node, but while investigating, we noticed this anomaly:

It seems the UMD module gets builded with webpack, while I can see that you are distributing ES and CJS format.
Inspecting at your package.json, it seems you are not defining the module, is it possible to release a version that fix the module as we rather not ship all of algoliasearch to our web clients.
Thanks in advance !

This screenshots illustrate the potential gain of using CJS :

With UMD :

With CJS :


Missing API to create a new request

The Requests extension doesn't expose a method to create a new request (POST /request).
To create a new request, I currently need to write something like this:
from urllib.parse import urljoin
from osctiny import Osc
from osctiny.utils.xml import get_objectified_xml

osc = Osc(url=[…])
params = {"cmd": "create"}
body = (
    "<request>"
    "  <action type='submit'>"
    f"    <source project='{project}' package='{package}' rev='1'/>"
    f"    <target project='{target}' package='{package}'/>"
    "  </action>"
    f"  <description>foo bar baz</description>"
    "</request>"
)
response = osc.request(
    url=urljoin(osc.url, osc.requests.base_path),
    method="POST",
    data=body,
    params=params,
    raise_for_status=False,
)
return get_objectified_xml(response)

An additional create(…) method that takes the source project, the target project, the package name, an optional revision, and an optional description, would be useful.

Thanks for working on this @crazyscientist. One thing I'm not seeing in the new method is a way to set a description on the request (like <description>foo bar baz</description> in my example above). Can this be added?


(Attempt #2) Fix error message when vim-cursorword is installed

Hi!
New PR that solves the issue brought up in #21 without the repeated highlight check.
Understandable that this isn't an important edge case (feel free to close this also), but wanted to get your thoughts on this solution. cursorword is one of my favorites of the few dozen plugins I have, but also the only one that gives me an error every time I install it.
Notes:

This increases startup time from ~0.1 milliseconds to ~0.15 milliseconds on my machine.
Tested on Vim 7.4 & 8 without issues.

I fixed the behavior, please confirm.

Worked, thanks :)


Feature/1469

What does this pull request aim to achieve? 💯
resolves #1469
Please be concise and link any related issue(s):

[x] I have checked that no one else is working on similar changes.
[x] I have read the latest README and understand the development workflow.
[x] The name of this branch is: feature/<branch_name>, or hotfix/<branch_name> if this is a hotfix. The branch is created in a cloned version of this repo, not a forked repo.
[ ] This branch is rebased/merged with the latest staging (or the latest production if it is a hotfix).
[x] There are no merge conflicts.
[x] There are no console warnings and errors.
[x] On the right hand side of this PR, I have linked the appropriate issue(s) under "Issues" and added a project progress under "Projects".

Thanks @Serhazor
This definitely solves @lmpeixoto 's issue.
However the issue @CamilaArias reported - I can't recreate in staging but I see it happening here (Chrome devtools - iPhone 6/7/8).
What specific device/browser were you using? I can suggest some improvement's to handle both (my main worry is this is making it worse for narrow mobile devices so I don't want to approve yet).

@Serhazor
Hi
May I re-run checks? I see that last Deploy Review Build started 10 days ago and there is Cancel workflow displayed.

This issue is no longer valid as the the 'Type' and 'Industry' dropdown' for 'Add/EDIT Org', are fixed via #1465 . hence, closing this issue as od now, please open a new issue, if this issue is observed again. Thanks

This issue is no longer valid as the the 'Type' and 'Industry' dropdown' for 'Add/EDIT Org', are fixed via #1465 . hence, closing this issue as od now, please open a new issue, if this issue is observed again. Thanks


Can no longer even import confluent-kafka after upgrading 1.8.2 > 1.9.0 (Apple M1 pro)

Description
upgrading 1.8.2 > 1.9.0 broke my code and I am unsure why. I am no longer even able to import confluent-kafka
context:
machine running the code :

kafka is located in AWS running through MSK (version 2.6.2)
not using any other confluent services or libraries other than confluent-kafka
How to reproduce
upgrade to 1.9.0 and import confluent-kafka
here is some terminal output (with some paths censored and spacing of commands)
(python-3.8.13) $ pip list
Package         Version
--------------- -------
confluent-kafka 1.8.2
Faker           13.13.0
pip             22.1.2
python-dateutil 2.8.2
setuptools      56.0.0
six             1.16.0

(python-3.8.13) $ python3
Python 3.8.13 (default, May  2 2022, 11:54:35)
[Clang 13.1.6 (clang-13<IP_ADDRESS>.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import confluent_kafka
>>> exit()

(python-3.8.13) $ pip uninstall confluent_kafka
Found existing installation: confluent-kafka 1.8.2
Uninstalling confluent-kafka-1.8.2:
  Would remove:
    /path/to/python/install/python-3.8.13/LICENSE.txt
    /path/to/python/install/python-3.8.13/lib/python3.8/site-packages/confluent_kafka-1.8.2-py3.8.egg-info
    /path/to/python/install/python-3.8.13/lib/python3.8/site-packages/confluent_kafka/*
Proceed (Y/n)? y
  Successfully uninstalled confluent-kafka-1.8.2

(python-3.8.13) $ pip install confluent_kafka
Collecting confluent_kafka
  Using cached confluent-kafka-1.9.0.tar.gz (110 kB)
  Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for confluent_kafka, since package 'wheel' is not installed.
Installing collected packages: confluent_kafka
  Running setup.py install for confluent_kafka ... done
Successfully installed confluent_kafka-1.9.0

(python-3.8.13) $ pip list
Package         Version
--------------- -------
confluent-kafka 1.9.0
Faker           13.13.0
pip             22.1.2
python-dateutil 2.8.2
setuptools      56.0.0
six             1.16.0

(python-3.8.13) $ python3
Python 3.8.13 (default, May  2 2022, 11:54:35)
[Clang 13.1.6 (clang-13<IP_ADDRESS>.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import confluent_kafka
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/python/install/python-3.8.13/lib/python3.8/site-packages/confluent_kafka/__init__.py", line 19, in <module>
    from .deserializing_consumer import DeserializingConsumer
  File "/path/to/python/install/python-3.8.13/lib/python3.8/site-packages/confluent_kafka/deserializing_consumer.py", line 19, in <module>
    from confluent_kafka.cimpl import Consumer as _ConsumerImpl
ImportError: dlopen(/path/to/python/install/python-3.8.13/lib/python3.8/site-packages/confluent_kafka/cimpl.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_rd_kafka_AclBindingFilter_new'

Checklist
Please provide the following information:

[x] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
[ ] Confluent.Kafka nuget version.
[x] Apache Kafka version.
[ ] Client configuration.
[x] Operating system.
[ ] Provide logs (with "debug" : "..." as necessary in configuration).
[ ] Provide broker log excerpts.
[ ] Critical issue.

please ask the same question in https://github.com/confluentinc/confluent-kafka-python


Upgrade pytest to version 7.2.0

pytest 5.4.3 isn't compatible with Python 3.10 and newer. Python 3.10 support was officially added in pytest 6.2.5, while Python 3.11 isn't officially supported yet but appears to work correctly as of pytest 7.2.0. Upgrade to version 7.2.0 and pack its new dependencies into the pytest bootstrap pex.
This also requires upgrading coverage from 5.5 to 6.5.0, which drops support for Python 3.5 and 3.6. (Both versions are now end-of-life.)
Fixes https://github.com/thought-machine/please/issues/2570.

This also causes //test:python2_coverage_test to fail, since coverage 5.0.0 drops support for Python 2. @samwestmoreland is this a problem?

We did have some users on Python 2 fairly recently, but they weren't using the plugin AFAIK.
I think we have to drop it at this point - it's more important we support 3.10 and 3.11 than the EOL versions. Although probably that should constitute a major version of the plugin?

Nice one, thanks, Chris! Yeah, I think more users will be happy that we're supporting pythons 3.10 and 3.11 than will be disappointed that we're dropping older versions. But regardless, I agree with Pebers - more important to support 3.10 and 3.11 and to stay up-to-date in general, especially in this instance given the performance gains.
The only question is over the release... We've been developing on the v1 branch and tagging the v1 releases from there. I think the thing to do is to merge the v1 branch back into master, and tag v2 on the master branch. Does anyone have any thoughts/objections?

Completely missed the activity on the v1 branch, sorry. I can rebase this on that branch if you like.

I'll copy your other fix over to v1 here: https://github.com/please-build/python-rules/pull/85

This is already on the v1 branch, which has been merged back into master now.


document all _internal values

Current list from a clustered data node:
OSS list:
Connected to http://localhost:8086 version 1.0.0
InfluxDB shell version: 1.0.0
> use _internal
Using database _internal
> show field keys
name: cq
--------
fieldKey    fieldType
queryFail   integer
queryOk     integer

name: database
--------------
fieldKey    fieldType
numMeasurements integer
numSeries   integer

name: httpd
-----------
fieldKey        fieldType
authFail        integer
clientError     integer
cqReq           integer
pingReq         integer
pointsWrittenFail   integer
pointsWrittenOK     integer
queryReq        integer
queryReqDurationNs  integer
queryRespBytes      integer
req         integer
reqActive       integer
reqDurationNs       integer
serverError     integer
statusReq       integer
writeReq        integer
writeReqActive      integer
writeReqBytes       integer
writeReqDurationNs  integer

name: queryExecutor
-------------------
fieldKey    fieldType
queriesActive   integer
queryDurationNs integer

name: runtime
-------------
fieldKey    fieldType
Alloc       integer
Frees       integer
HeapAlloc   integer
HeapIdle    integer
HeapInUse   integer
HeapObjects integer
HeapReleased    integer
HeapSys     integer
Lookups     integer
Mallocs     integer
NumGC       integer
NumGoroutine    integer
PauseTotalNs    integer
Sys     integer
TotalAlloc  integer

name: shard
-----------
fieldKey    fieldType
diskBytes   integer
seriesCreate    integer
writeBytes  integer
writePointsErr  integer
writePointsOk   integer
writeReq    integer
writeReqErr integer
writeReqOk  integer

name: subscriber
----------------
fieldKey    fieldType
pointsWritten   integer
writeFailures   integer

name: tsm1_cache
----------------
fieldKey        fieldType
WALCompactionTimeMs integer
cacheAgeMs      integer
cachedBytes     integer
diskBytes       integer
memBytes        integer
snapshotCount       integer

name: tsm1_engine
-----------------
fieldKey            fieldType
cacheCompactionDuration     integer
cacheCompactions        integer
tsmFullCompactionDuration   integer
tsmFullCompactions      integer
tsmLevel1CompactionDuration integer
tsmLevel1Compactions        integer
tsmLevel2CompactionDuration integer
tsmLevel2Compactions        integer
tsmLevel3CompactionDuration integer
tsmLevel3Compactions        integer

name: tsm1_filestore
--------------------
fieldKey    fieldType
diskBytes   integer
numFiles    integer

name: tsm1_wal
--------------
fieldKey        fieldType
currentSegmentDiskBytes integer
oldSegmentsDiskBytes    integer
writeErr        integer
writeOk         integer

name: write
-----------
fieldKey    fieldType
pointReq    integer
pointReqLocal   integer
req     integer
subWriteDrop    integer
subWriteOk  integer
writeDrop   integer
writeError  integer
writeOk     integer
writeTimeout    integer

>

Cluster-specific values:
Connected to http://<IP_ADDRESS>:8086 version 1.0.0-c1.0.0
InfluxDB shell version: v1.0.0
> use _internal
Using database _internal
> show field keys
name: cluster
-------------
fieldKey        fieldType
copyShardReq        integer
createIteratorReq   integer
expandSourcesReq    integer
fieldDimensionsReq  integer
removeShardReq      integer
writeShardFail      integer
writeShardPointsReq integer
writeShardReq       integer

name: cq - same
name: database - same

name: hh
--------
fieldKey        fieldType
writeShardReq       integer
writeShardReqPoints integer

name: hh_processor
------------------
fieldKey        fieldType
bytesRead       integer
bytesWritten        integer
queueDepth      integer
writeNodeReq        integer
writeNodeReqFail    integer
writeNodeReqPoints  integer
writeShardReq       integer
writeShardReqPoints integer

name: httpd - same
name: queryExecutor - same
name: runtime - same
name: shard - same
name: subscriber - same
name: tsm1_cache - same
name: tsm1_engine - same
name: tsm1_filestore - same
name: tsm1_wal - same

name: write - mostly the same, three new fields, no writeDrop field
-----------
fieldKey    fieldType
pointReq    integer
pointReqHH  integer
pointReqLocal   integer
pointReqRemote  integer
req     integer
subWriteDrop    integer
subWriteOk  integer
writeError  integer
writeOk     integer
writePartial    integer
writeTimeout    integer

>

https://github.com/influxdata/docs.influxdata.com/blob/master/content/influxdb/v1.0/troubleshooting/internal_statistics.md and https://github.com/influxdata/docs.influxdata.com/blob/master/content/enterprise/v1.0/troubleshooting/internal_statistics.md created to house the info.

WIP on branch https://github.com/influxdata/docs.influxdata.com/tree/mr-internal-stats documents the OSS stats. I should have time tomorrow to finish up the Enterprise stats.

Great work, Mark, thanks! That looks like exactly what we need!

That branch is now in PR #777, and I've opened a separate placeholder issue at #780 to track other stats changes slated for 1.1.

Hey,
Where are you on this matter? I see that you've made good progress on this PR https://github.com/influxdata/docs.influxdata.com/pull/777 but unfortunately the current documentation (v1.2) for OSS and entreprise have not a good documentation for values stored in the _internal measurements.
The internal monitoring blog post is old (Sept 2015) and is not complete enough compared to what has been added recently.

So is this the most current documentation on these values?

Closing -- will not be documenting _internal metrics. Intended for internal / support use.

Actually, these should be documented. Not in the context of the _internal database, but in the context of the InfluxDB Telegraf input plugin. The stats that come from that are the same stats that go to _internal and they're what a user would want to pay attention to  if they want to monitor their production database.

Closing -- documentation added for _internal/telegraf measurement metrics (http://docs.influxdata.com/platform/monitoring/tools/measurements-internal/). Next docs will be added explaining use with _internal for development, using Telegraf for external monitoring added in http://docs.influxdata.com/platform/monitoring/.


Pre-commit hook to use clang-format

Hello! I am proposing to use clang-format as a pre-commit hook.
For this I used the pre-commit micro framework, but I can remove it and use simple 'cp' commands if you think would be better. I also added usage details in the readme.
For this clang-format configuration I used the BasedOnStyle:  Google, but it was just to initialize, we can discuss here which style base is better or just create a custom one. There is also a clang-format config detector that analyses the code and create the configurations but it is a windows-based software and I'm not able to try it now.

Let us try to merge this and see what happens.

merged

I'm -1 on having pre-commit hooks, since it moves the requirement of running the checks to the developer. Right now, there isn't any way for us to verify that the pull request sent to ada/idna is compliant with our clang-format. We should have a github workflow for validation purposes.

I like pre-commit hooks because it kinda ensure that things happened before I commit. But yeah, it is easily bypassed or just ignored.
@anonrig So do you think pre-commit should be removed + add a workflow to check the code style, or just add a workflow  to check the code style?

I think pre-commit should be a recommendation to reduce CI errors, but we should definitely have a GitHub workflow hook if we want to make sure we are following the formatting guidelines.

Sounds good!