record_transformer ignores all but first transformation

I'm trying to use the record_transformer plugin to add a few fields to a record, but only the first one I specify gets applied. This is especially strange since I have an existing configuration where this is not the case. I can't figure out what I'm doing wrong.

fluentd or td-agent version: fluentd-0.12.40
Environment information: Modified Amazon Linux
Configuration: The relevant portions are the two chunks shown in this gist (sorry, I was unable to get the formatting to work well in this issue.
The problem: The fields listed in the first filter all get applied properly. In the second filter, the fluent_tag field gets properly set - I can see that coming through in my log aggregator - but no other fields that I specify there get set. Ultimately, I'm trying to convert the "message" field in my record to JSON, and I've tried a few things to that end, but even if I do a plain, hardcoded string like in the example config in the gist, that data does not come through at all. I have ramped logging up to DEBUG level, and the td-agent output has nothing interesting in it. I'm sure I could solve my JSON issue if I could get td-agent to pay attention to the full extent of this config.

I tested following configuration based on your gist.
<source>
  @type dummy
  tag system.local6.debug
</source>

<filter **.**>
  @type record_transformer
  <record>
    fluent_tag ${tag}
    env stage
    fluentd_instance_id i-abc1234
  </record>
</filter>

<filter system.local6.debug>
  @type record_transformer
  <record>
    fluent_tag bashlogs
    literally_anything else_here_gets_ignored
  </record>
</filter>

<match>
  @type stdout
</match>

Result:
2018-03-29 11:19:10 +0900 [info]: reading config file path="local/1914.conf"
2018-03-29 11:19:10 +0900 [info]: starting fluentd-0.12.40
2018-03-29 11:19:10 +0900 [info]: gem 'fluentd' version '0.12.40'
2018-03-29 11:19:10 +0900 [info]: adding filter pattern="**.**" type="record_transformer"
2018-03-29 11:19:10 +0900 [info]: adding filter pattern="system.local6.debug" type="record_transformer"
2018-03-29 11:19:10 +0900 [info]: adding match pattern="**" type="stdout"
2018-03-29 11:19:10 +0900 [info]: adding source type="dummy"
2018-03-29 11:19:10 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type dummy
    tag system.local6.debug
  </source>
  <filter **.**>
    @type record_transformer
    <record>
      fluent_tag ${tag}
      env stage
      fluentd_instance_id i-abc1234
    </record>
  </filter>
  <filter system.local6.debug>
    @type record_transformer
    <record>
      fluent_tag bashlogs
      literally_anything else_here_gets_ignored
    </record>
  </filter>
  <match>
    @type stdout
  </match>
</ROOT>
2018-03-29 11:19:11 +0900 system.local6.debug: {"message":"dummy","fluent_tag":"bashlogs","env":"stage","fluentd_instance_id":"i-abc1234","literally_anything":"else_here_gets_ignored"}
2018-03-29 11:19:12 +0900 system.local6.debug: {"message":"dummy","fluent_tag":"bashlogs","env":"stage","fluentd_instance_id":"i-abc1234","literally_anything":"else_here_gets_ignored"}
2018-03-29 11:19:13 +0900 system.local6.debug: {"message":"dummy","fluent_tag":"bashlogs","env":"stage","fluentd_instance_id":"i-abc1234","literally_anything":"else_here_gets_ignored"}
2018-03-29 11:19:14 +0900 system.local6.debug: {"message":"dummy","fluent_tag":"bashlogs","env":"stage","fluentd_instance_id":"i-abc1234","literally_anything":"else_here_gets_ignored"}
2018-03-29 11:19:15 +0900 system.local6.debug: {"message":"dummy","fluent_tag":"bashlogs","env":"stage","fluentd_instance_id":"i-abc1234","literally_anything":"else_here_gets_ignored"}

Is this expected result?
Could you show me full configuration like the above and actual input and expected result?

My full config can be found here, with a few redactions. The records I'm particularly interested in here are produced by a command logger on our systems. Every system sends records of all commands run up through rsyslog (hence the local6.debug tag). I want the fluent_tag (which serves as an identifier in our environments) to be more meaningful than "local6.debug", so I overwrite that here with "bashlogs". But due to a complication in Loggly where this field is ordinarily a non-JSON string, I want to rewrite the full record, which is valid JSON, as a different field name. This will cause Loggly to pick up on the fact that it is JSON and make each of those JSON fields a searchable, indexable value. Here's an example of a record:
{"user": "mylinuxusername", "pid": 1234, "command": "echo 'blah'", "exit_code": 0}

Now, that comes through in Loggly along with a timestamp and some other data as a field called "message". I believe that at the point where I'm running this filter that it is actually the full record and the "message" bit gets applied by Loggly. So what I'd like to do is use record transformer to do something like
message_json = ${message}

Or maybe, with enable_ruby
message_json = ${record.to_json}

The details of converting to JSON are almost pointless, though, because I can't even get td-agent to attach a static string using something like
field value

inside of a <record> block.

I've found an alternative solution in a feature of Loggly, so I'm up and running now. I'm still baffled by this problem, though. I'll close the issue because I personally don't need an answer any longer. However, the configurations I've linked still have the problem described, if that's helpful to anybody else.


[WIP] Offboard updates after offboarding John

This is not ready to merge (I want to finish rotating the keys to see if any other tasks fall out of that work) but in addition to my minor edits I reorganized the checklist based on the role needed to do the task.
I think we'd complete checklists faster if it was clear to the assignee who to work with to get each task done.
WDYT?  @mogul @brittag

(Leaving unmerged, but I approve everything so far.)

Looks good to me too!


Investigate the 10x multi-threaded perf improvement hiding in distribution/scheduling.

Found a great example of memory system effects between single- (read=1 core) and multi-threaded (yellow=32 cores) with cumulative time (ignore total times as they are two benchmark runs with different counts, mean/median is relevant):

(this is with allocator zero page optimizations disabled so that there are no during execution faults)
This scales with worker count; so here's red=1 core/yellow=8 cores:

Without tracy, 1 core / 8 cores / 32 cores:
--------------------------------------------------------------------------
Benchmark                                Time             CPU   Iterations
--------------------------------------------------------------------------
BM_learn/process_time/real_time      23854 ms        23849 ms            6   // 1
BM_learn/process_time/real_time       4776 ms        29411 ms           29   // 8
BM_learn/process_time/real_time       2684 ms        46996 ms           52   // 32

I believe this is either contention or cache locality issues but have not dug in to see on what (which loads/stores are bad). It could also be memory bandwidth saturation. For 12MB buffers it's definitely putting pressure on the cache hierarchy:
CPU Caches:
  L1 Data 32 KiB (x32)
  L1 Instruction 32 KiB (x32)
  L2 Unified 512 KiB (x32)
  L3 Unified 16384 KiB (x8)

What this hints at is that we run 10x slower/tile on 32x more threads, so we end up with ~3x lower latency overall for this particular dispatch but do so by being 10x more wasteful than we could be if we were perfectly scaling.
I picked this dispatch because it's a fairly mundane elementwise + transpose:
#map0 = affine_map<(d0, d1) -> (d0, d1)>
#map9 = affine_map<(d0, d1) -> (d1, d0)>
  stream.executable private @__inference_learn_231830_dispatch_2094 {
    stream.executable.export public @__inference_learn_231830_dispatch_2094
    builtin.module {
      func.func @__inference_learn_231830_dispatch_2094(%arg0: !stream.binding {stream.alignment = 64 : index}, %arg1: !stream.binding {stream.alignment = 64 : index}, %arg2: !stream.binding {stream.alignment = 64 : index}, %arg3: i32, %arg4: i32, %arg5: i32 {stream.values = [36954688 : i32, 237015040 : i32, 327192576 : i32, 417370112 : i32, 507547648 : i32, 597725184 : i32, 687902720 : i32, 778080256 : i32, 868257792 : i32, 958435328 : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32,<PHONE_NUMBER> : i32, -2074046464 : i32, -2040492032 : i32]}, %arg6: i32 {stream.values = [212684864 : i32, 246255680 : i32, 279826496 : i32, 313397312 : i32, 346968128 : i32, 380538944 : i32, 414109760 : i32, 447680576 : i32, 481251392 : i32, 514822208 : i32, 548393024 : i32, 581963840 : i32, 615534656 : i32, 649105472 : i32, 682676288 : i32, 716247104 : i32, 749817920 : i32, 783388736 : i32, 816959552 : i32, 850530368 : i32, 884101184 : i32, 917672000 : i32, 951242816 : i32, 984813632 : i32]}) {
        %cst = arith.constant 0.00999999977 : f32
        %c32_i64 = arith.constant 32 : i64
        %0 = arith.extui %arg3 : i32 to i64
        %1 = arith.extui %arg4 : i32 to i64
        %2 = arith.shli %1, %c32_i64 : i64
        %3 = arith.ori %0, %2 : i64
        %4 = arith.index_cast %3 : i64 to index
        %5 = arith.extui %arg5 : i32 to i64
        %6 = arith.index_cast %5 : i64 to index
        %7 = arith.extui %arg6 : i32 to i64
        %8 = arith.index_cast %7 : i64 to index
        %9 = stream.binding.subspan %arg0[%4] : !stream.binding -> !flow.dispatch.tensor<readonly:1024x3072xf32>
        %10 = stream.binding.subspan %arg1[%6] : !stream.binding -> !flow.dispatch.tensor<readonly:3072x1024xf32>
        %11 = stream.binding.subspan %arg2[%8] : !stream.binding -> !flow.dispatch.tensor<writeonly:1024x3072xf32>
        %12 = flow.dispatch.tensor.load %9, offsets = [0, 0], sizes = [1024, 3072], strides = [1, 1] : !flow.dispatch.tensor<readonly:1024x3072xf32> -> tensor<1024x3072xf32>
        %13 = flow.dispatch.tensor.load %10, offsets = [0, 0], sizes = [3072, 1024], strides = [1, 1] : !flow.dispatch.tensor<readonly:3072x1024xf32> -> tensor<3072x1024xf32>
        %14 = linalg.init_tensor [1024, 3072] : tensor<1024x3072xf32>
        %15 = linalg.generic {indexing_maps = [#map0, #map9, #map0], iterator_types = ["parallel", "parallel"]} ins(%12, %13 : tensor<1024x3072xf32>, tensor<3072x1024xf32>) outs(%14 : tensor<1024x3072xf32>) {
        ^bb0(%arg7: f32, %arg8: f32, %arg9: f32):
          %16 = arith.mulf %arg8, %cst : f32
          %17 = arith.subf %arg7, %16 : f32
          linalg.yield %17 : f32
        } -> tensor<1024x3072xf32>
        flow.dispatch.tensor.store %15, %11, offsets = [0, 0], sizes = [1024, 3072], strides = [1, 1] : tensor<1024x3072xf32> -> !flow.dispatch.tensor<writeonly:1024x3072xf32>
        return
      }
    }
  }

Input MHLO model: https://storage.googleapis.com/iree-shared-files/nod-perf/bert_large_250M.mlir
Compilation flags:
--iree-stream-resource-index-bits=64
--iree-vm-target-index-bits=64
--iree-hal-target-backends=dylib-llvm-aot
--iree-llvm-target-triple=x86_64
--iree-llvm-target-cpu=host
--iree-llvm-target-cpu-features=host
--iree-codegen-llvm-number-of-threads=32

Run flags:
--task_topology_group_count=32
--function_input=1x512xi32=0
--function_input=1x512xi32=0
--function_input=1x512xi32=0
--function_input=1xi32=0

Added to buffer_heap.c iree_hal_heap_buffer_create:
    memset(buffer->data.data, 0xCD, buffer->data.data_length);

Next steps are to get perf counters to verify what's causing the 10x difference (cache misses, pending memory ops, something else spooky, etc), isolate this into a microbenchmark, and try it on some different systems.

are --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 default now ? Or would / should they become default ?

comment withdrawn :D

Some of this may be thread migration and lock contention (though definitely not all). On Windows I'm noticing that tracy with fiber tracing enabled causes an excessive number of context switches as the shared tracy lock is contended - it's bad enough that I don't think we can use tracy fibers by default right now if we also want any kind of performance investigation. Even with tracy off entirely, though, I still see a great deal more switches than I expected. Almost all are coming from locks and in particular the one in iree_task_queue_try_steal and the main executor lock in iree_task_executor_coordinate. Nearly every time work is stolen or the executor coordination is called on a worker we switch cores:

Our workers dancing around cores:

By just changing the coordination lock and try-steal source locks to try-locks quite a few drop out:

It should look like this, which is init in a model with fake weights that does not require any coordination as it runs:

Making the iree_task_queue_t wait-free would help a lot as would better executor coordination behavior. A quick peek at tracy's lock visualization shows the iree_atomic_slist_t (which isn't atomic yet 🤦) as being the most heavily contended lock, and iree_task_pool_release shows as being fairly contended as well (makes sense as shard tasks cycle through it from all threads).

@benvanik Still open? Still P1?


feat(pie-form-label): DSW-1585 update pie-form-label to work with form components

Describe your changes (can list changeset entries if preferable)
Author Checklist (complete before requesting a review)

[ ] I have performed a self-review of my code
[ ] If it is a core feature, I have added thorough tests
[ ] If it is a PIE Docs change, I have reviewed the Docs site preview
[ ] If it is a component change, I have reviewed the Storybook preview
[ ] If there are visual test updates, I have reviewed them properly before approving

Reviewer checklists (complete before approving)
Reviewer 1

[ ] If it is a PIE Docs change, I have reviewed the PR preview
[ ] If there are visual test updates, I have reviewed them

Reviewer 2

[ ] If it is a PIE Docs change, I have reviewed the PR preview
[ ] If there are visual test updates, I have reviewed them

/snapit

/snapit

/snapit


Error when reading images with very small file size

The function find_loader() called by simage_check_supported() returns NULL when called on images with a very small file size (<256 bytes). Example images for this are the test.gif (35 bytes) and test.png (82 bytes) files included in Homebrew.
When using the loaders test, this results in a failed test and the message "x is not in a supported format":
$ loaders test.gif
``test.gif'' is not in a supported format
$ loaders test.png
``test.png'' is not in a supported format

This is due to the feof(fp) call in simage.c. As the identity_func() gets the readlen value as parameter, this can likely be omitted.
https://github.com/coin3d/simage/blob/3940692136e02b9e62b3061cf72f053827b5494e/src/simage.c#L159-L160

Thanks for the thorough bug report. I just committed the change and added  the two small files to the tests. I'll publish a 1.8.3 release later today.

Thank you for the quick fix and the new release.


The Songs Week 6 Was Has Been Crashed

so I Test The Other Songs And Stage I Check To Look Like are Fine
The Problem IS Week 6 Cannot Playable IDK Why? is Keep Crashed

use latest source probably a noteskin issue


Execution failed for task ':react-native-ble-plx:compileReleaseAidl'.

When I try to add the library to an existing ignite project and try to run via react-native run-android I get Execution failed for task ':react-native-ble-plx:compileReleaseAidl'.
I looked around and the common suggestion is to upgrade the build tools version but react native wants that fixed.  Any suggestions?

I have the same issue!

What went wrong:
Execution failed for task ':react-native-icons:compileReleaseAidl'.

java.lang.IllegalStateException: aidl is missing

Please retest with latest version and reopen if needed.

@Cierpliwy  I am still getting the same error with the latest version too. Please re-open this if not fixed.


Find way to deprecate AWSShape members

Currently deprecated members are removed. But if we keep them around and add a `@available(*, deprecate, ...) to deprecate them, you get a warning in the init() function where it is initialised. Need a better solution

Could create two init() functions. One with deprecated members in and one without


Compute tang key directly without doing frankenstein ECMR computation

Current version implements the first step of EMCR handshake directly
and then for the second part it uses jwx ECDH decryption code.
Let's compute the encryption key directly and then just use DIRECT
decryption the same as with other pins.

cc @lack for a review


Integrate with React Native NavigatorIOS

More of a question than an issue, but was it considered contributing this back to the React Native NavigatorIOS component?

I don't know React Native, so I didn't consider that. Feel free to do this by yourself, though ⌨.


Added annotations to zerver/views/report.py

Automated message from Dropbox CLA bot
@bkeyvani, thanks for the pull request! It looks like you haven't yet signed the Dropbox CLA. Please sign it here.

Posted one comment, otherwise this looks great @bkeyvani!  If you sign the CLA and fix that small issue, I can merge this.

Automated message from Dropbox CLA bot
@bkeyvani, thanks for signing the CLA!

@timabbott thanks for the comments. I updated the type definitions for the Dict type as you had suggested. I also had to change the type of the stacktrace from str to unicode since the tests were throwing the following error:
zerver/views/report.py:86: error: Incompatible types in assignment (expression has type "unicode", variable has type "str")
The tests do pass with the current pull request, but please let me know if this is OK or if any additional changes are needed.

You should use six.text_type instead of unicode as we're aiming for python 3 compatibility and python 3 doesn't have unicode.
We generally do from six import text_type and use text_type in annotations instead of doing import six and using six.text_type in annotations.
We should be using text_type and str carefully. Python 2 implicitly converts str and unicode using the 'ascii' codec and non-ascii characters can lead to exceptions. To deal with this problem, the approach I'm following (I don't think we have any formal consensus on this in Zulip right now) is to use text_type in as many places as possible except where it's absolutely necessary to use str (like dict unpacking). We can't do the opposite because there are too many sources of text_type, like all CharFields in Django.
Also mypy doesn't complain if we pass an str to a function which expects a text_type, but the reverse is not allowed.

Thanks for the tip @sharmaeklavya2 . Sorry for the delay, I was under the weather the past few days. I made the change and it is now passing the test. Also, I wasn't sure from your comment if you wanted me to also change the str types to text_type or leave them as is, since they seem to be passing the test for now. If prefer those to be changed to text_type for future proofing just let me know and I'll update this PR.

No worries for the delay.
Sorry I should have been more explicit - please change str to text_type where it's relevant.

Thanks @sharmaeklavya2 . Changed the type for the parameters that you had mentioned to text_type.
Also would you please elaborate more on what you mean by this (what do you mean by "do the opposite"?):

We can't do the opposite because there are too many sources of text_type, like all CharFields in Django

Best,
B

I meant that trying to annotate all strings as str and annotating strings as text_type only when absolutely necessary would be problematic. The reason I gave for this in my earlier comment is that most strings in Zulip are text_type, so there is less chance of conflict. That is not a good reason.
Sometimes I was myself a bit confused on how to decide between str and text_type. It was more difficult when someone asked me for help on this topic. So I gave it a lot of thought and that led to a new section in the mypy docs about annotating strings. I got this reviewed from @timabbott.

I think your pull request is ready to merge. @timabbott will probably ask you to squash your commits.
Thanks for working on this @bkeyvani !

Yep, this looked good; I went ahead and squashed the commits and merged this as ad895eb690f9c045a4d9a3abc69fa30ce7cf1f2c.  Thanks @bkeyvani and @sharmaeklavya2!

Thank you both @sharmaeklavya2 & @timabbott :) I really enjoyed this and look forward to contributing more to your library :)
Best,
B


fix: make input of .required() readonly

More info: https://github.com/trpc/trpc/issues/5538
Unsure if this is a legit fix. I could see this working as designed.
cc @tkdodo @PeterMK85

This has landed in Zod 3.23.
https://github.com/colinhacks/zod/releases/tag/v3.23.0


[FAB-18401] Remove unreachable and unnecessary code in gossip membership

In gossip, alive messages are stored in membership stores, and they also have logical timestamps.
When a peer learns of an alive message with a higher logical timestamp of its latest alive message (for a specific peer),
it enters learnExistingMembers() and tentatively updates the latest envelope and gossip message of what is stored in the membership store.
However, for completeness, the code checks if the membership store already contains an earlier instance, and if not, populates the membership store with a new instance.
This population is not needed, because the code only enters the function if it already knows the peer, hence the membership
store contains an earlier instance.
This commit simply removes the unneeded code path where the instance is unknown despite the fact that we know about the peer already.
Change-Id: Ia91cb35eabe18a8771ad12a30b06489892a2b49f
Signed-off-by: Yacov Manevich<EMAIL_ADDRESS>

@Mergifyio backport release-2.3

@Mergifyio backport release-2.3

@Mergifyio backport release-2.2

@Mergifyio backport release-2.2

@Mergifyio backport release-1.4

@Mergifyio backport release-1.4


Using Docker to run in standalone mode, data upload failed.

System information

Have I written custom code (yes/no): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): centos7
FATE Flow version (use command: python fate_flow_server.py --version): {'FATE': '1.11.2', 'FATEFlow': '1.11.1', 'FATEBoard': '1.11.1', 'EGGROLL': '2.5.1', 'CENTOS': '7.2', 'UBUNTU': '16.04', 'PYTHON': '3.8', 'MAVEN': '3.6.3', 'JDK': '8', 'SPARK': '3.4.0'}
Python version (use command: python --version): Python 3.8.13

Describe the current behavior
The container star command is as follows.
$ docker run -d -it \
    --name single_fate \
    --restart=always \
    -p 8080:8080 \
    -p 9380:9380 \
    federatedai/standalone_fate:1.11.2

$ docker ps |grep single_fate
cc2b7babbc26        federatedai/standalone_fate:1.11.2              "./bin/docker-entryp…"   22 minutes ago      Up 22 minutes          <IP_ADDRESS>:9380->9380/tcp, <IP_ADDRESS>:9090->8080/tcp                                                       single_fate

I followed the tutorial and ran the code for Pipeline tutorial upload, but upload data failed on this code,
pipeline_upload.upload(drop=1)

error info is :
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/miniconda3/envs/fate/lib/python3.8/site-packages/pipeline/utils/invoker/job_submitter.py:61, in JobInvoker.upload_data(self, submit_conf, drop)
     60 if 'retcode' not in result or result["retcode"] != 0:
---> 61     raise ValueError
     63 if "jobId" not in result:

ValueError:

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[17], line 1
----> 1 pipeline_upload.upload(drop=1)

File ~/miniconda3/envs/fate/lib/python3.8/site-packages/loguru/_logger.py:1251, in Logger.catch.<locals>.Catcher.__call__.<locals>.catch_wrapper(*args, **kwargs)
   1249 def catch_wrapper(*args, **kwargs):
   1250     with catcher:
-> 1251         return function(*args, **kwargs)
   1252     return default

File ~/miniconda3/envs/fate/lib/python3.8/site-packages/pipeline/backend/pipeline.py:664, in PipeLine.upload(self, drop)
    662 upload_conf = self._construct_upload_conf(data_conf)
    663 LOGGER.debug(f"upload_conf is {json.dumps(upload_conf)}")
--> 664 self._train_job_id, detail_info = self._job_invoker.upload_data(upload_conf, int(drop))
    665 self._train_board_url = detail_info["board_url"]
    666 self._job_invoker.monitor_job_status(self._train_job_id,
    667                                      "local",
    668                                      0)

File ~/miniconda3/envs/fate/lib/python3.8/site-packages/pipeline/utils/invoker/job_submitter.py:69, in JobInvoker.upload_data(self, submit_conf, drop)
     67     data = result["data"]
     68 except BaseException:
---> 69     raise ValueError("job submit failed, err msg: {}".format(result))
     70 return job_id, data

Describe the expected behavior
upload data success.
Contributing

Do you want to contribute a PR? (yes/no):  no
Briefly describe your candidate solution(if contributing):

When I use the curl command inside a container.：
curl http://<IP_ADDRESS>:9380/
{"retcode":100,"retmsg":"<NotFound '404: Not Found'>"}

When I use the curl command outside of the container.：
$ curl http://<IP_ADDRESS>:9380/
curl: (56) Recv failure: Connection reset by peer

So I guess it's a problem with the default host.

After I changed the host in service_conf.yaml of fateflow to <IP_ADDRESS>, the data was uploaded successfully.
The modified configuration is as follows:：
fateflow:
  # you must set real ip address, <IP_ADDRESS> and <IP_ADDRESS> is not supported
  host: <IP_ADDRESS>
  http_port: 9380
  grpc_port: 9360
  # when you have multiple fateflow server on one party,
  # we suggest using nginx for load balancing.
  nginx:
    host:
    http_port:
    grpc_port:
  # use random instance_id instead of {host}:{http_port}
  random_instance_id: false

Thank you very much for your feedback. You are correct, and we will work on optimizing this issue in the future.


Private registries limits (repo and tag)

Hi,
I noticed that the repository limit (repo_limit) and tag limit (tag_limit) were not applied for private registries. So I went to read the code and it seems like it has been implemented for public registries only.
I'd just like to know if this is a choice or if the implementation is missing?

You are right. This only applies to docker public registry, to limit number of images to be scanned. Other registries don't have this limit.


Update to 0.3.2

This module should probably be refactored to use Hiera, but in the meantime I'm bumping to the latest stable version.

argh PR'ed to the wrong place :cry:


Remove ng-annotate and ng-filesort

ng-annotate is no longer maintained. Its replacement is based on Babel which we don't use. It brings in a lot of obsolete dependencies, and it forces us to avoid using some new JavaScript syntax.
And, apparently, it can be replaced by a regular expression.
The processed files are identical before and after the second commit.

Sorry, it's very hard to see what really is going on here with all the formatting changes. Is it possible to split those out into a separate commit, or alternatively, give some "reviewers' guide" to this PR?

Sorry, it's very hard to see what really is going on here with all the formatting changes. Is it possible to split those out into a separate commit, or alternatively, give some "reviewers' guide" to this PR?

If you look at the first commit, it's just adding angular.module('biggraph') everywhere instead of chaining. This results in some whitespace changes too, enforced by ESLint. You're right that it would have been better to have these in separate commits, sorry about that. But if you only look at this commit, maybe it's tolerable? GitHub is pretty helpful with the highlighting.

Just checking, what's up with this PR? Merge after some conflict resolution?

Just checking, what's up with this PR? Merge after some conflict resolution?

The tests are green. It can only mean MERGE!!!


small problem with list formatting/structure

PING @funilrys  !!!!!
I just noticed this and it looks like after new lists are generated the # End... line is not where it's suppose to be(at the bottom).
The issue is present only in that particular list for now but it's important for my scripts to work properly on my end to be at the very end.
Could you please look in to it whenever you find some free time. Thank you 👍

Thank you Steve @StevenBlack 👍
I'm not sure do you want this closed or not(yet)?


Update understanding-applocker-allow-and-deny-actions-on-rules.md

https://github.com/MicrosoftDocs/windows-itpro-docs/issues/4264
Added wording for clarification.

Hi @nenonix and @JohanFreelancer9 for copy review please. Thank you.

Hi @lindspea @nenonix and @JohanFreelancer9 from a perspective of a regular user reading the article, the amendments are still confusing for me. So basically, the default behavior is to 'deny action' unless explicitly added in the 'allowed list.' As stated in the article, the AppLocker rule collection functions as an allowed list of files, since by default, it's set to deny permission and you need to grant allow permission to override it. Anyway, looking forward to suggestions and updates. Thank you.

Not sure what I have missed out on regarding comma use in English grammar, but I would like to see the comma reinserted in the sentence
"Unlike Software Restriction Policies (SRP), each AppLocker rule collection functions as an allowed list of files."
To my mind, it should be indifferent whether the clause "unlike Software Restriction Policies (SRP)" is added at the end or beginning, as long as the comma is used as a separator between the clause and the main sentence. (I could be wrong, though.)

Hi @mjcaparas this pull request has been copy edited and is ready for your final review and approval. Please let us know if there are outstanding issues to be resolved prior to merge. Thank you. See: commits and changes.

Hi @mjcaparas thanks for approving this pull request. Can you also merge please? Thank you.

Thank you so much @mjcaparas for merging this PR.


Adding EFS volumes for cache, parquet and duckdb storage

Related to https://github.com/huggingface/datasets-server/issues/1407
Adding new volumes for: cache (datasets library), parquet and duckdb
Based on https://github.com/huggingface/infra/pull/607, the persistenceVolumeClaims should be:

datasets-server-cache-pvc
datasets-server-parquet-pvc
datasets-server-duckdb-pvc

I think it depends on https://github.com/huggingface/infra/pull/607/files to be merged/implemented first.

@rtrompier regarding datasets-server migration to EFS usage, not quite sure if the changes in this PR are enough, could you advice?

Nice. I'll let @rtrompier chime in, as it's really infra related. I think it would be good to test on staging before prod. Do we have dedicated EFS for staging?

Nice. I'll let @rtrompier chime in, as it's really infra related. I think it would be good to test on staging before prod. Do we have dedicated EFS for staging?

Yes we have EFS for staging env as well. I will create volume tomorrow for both environment.


Add Codeowners linter tool

This is the initial PR for the Azure.Sdk.Tools.CodeownersLinter tool.
What this tool does:
This tool will lint a CODEOWNERS file including our specific brand of Metadata. Note: The linter can handle the new metadata tags/format as well as the existing metadata. Our CODEOWNERS files contain more than source path/owner information, we also have metadata tags that are used in various workflows. All of these tags require more than one line in the CODEOWNERS file and these are referred to as blocks. _For example, a PRLabel entry needs to be part of a block that ends in a source path/owner line.
Note: Changes to the existing CodeownersParser are not functional outside of FileHelpers.LoadCodeownersFileAsStringList. The other changes were to declare monikers and separators as constants which are being used by the linter.
What changes will be required for every owner?

Owners, whether they're users or teams, need to have write permissions if they're going to be in a CODEOWNERS file. For teams this means that the team needs to be under the azure-sdk-write team. For owners this means they'll need to be under azure-sdk-write or in a team under azure-sdk-write. This is already true for any owner on a source path/owner line in CODEOWNERS but it also needs to be true for the metadata entries for our workflows.
Owners, specifically users, need to be public members of the Azure org. This was already true and the instructions for requesting Azure repository access clearly state this. The reason is that some workflows, like GitHub Actions processing, check Azure org membership and are unable to see Private members. This results in workflow processing that thinks a user isn't a member of Azure and is why some people file an issue and it gets labeled as Customer Reported even though they're an Azure org member.

What is verified during linting?

Owners

Users and Teams are verified to have write permissions. (see Owners above)
Users are also verified to be Public members of Azure.

Labels

Whether or not the label exists in a particular repository.

Source paths/Globs (not something GitHub verifies)

Does the path (directory or file) exist?
Does the glob contain forbidden characters?
Does the glob match anything in the repository?

Block formatting - There are several existing metadata tags and a couple more getting added or changed. Each tag is associated with an owner or label and those pieces are verified in their respective verifications. The block formatting verification ensures that things within a block are correct. Examples would be:

A PRLabel needs to be part of a block that ends in a source path/owner line
A ServiceLabel needs to be part of a block that ends in a source path/owner line OR needs to be immediately followed by a #/<NotInRepo>/ line with declared owners.

Where does the owner/label data come from?
The team/user, org visibility and repository label data are pre-populated by the github-team-user-store which runs daily as part of the pipeline-owners-extractor and is stored in blob data that workflows can use. This had to be done this way because of the specific permissions required to get the information from GitHub. This also means that some types of fixes, such as user changing their Azure org membership from private to public, won't be immediately picked up as the pipeline-owners-extractor pipeline would need to update the data in storage first.

Where will this run?
The goal here is to run this for at least the azure-sdk-for-<language> public and private repositories as both a nightly pipeline as well as a CI pipeline when CODEOWNERS is changed. Where and when we want to run this is the reason the data had to be in blob storage as we can't have tokens in public pipelines.
What about existing errors?
Even in repositories whose CODEOWNERS files are valid, according to GitHub, have errors. Because of this, repositories where this will be enabled will have a CODEOWNERS_baseline_errors.txt file which contains a deduped list of every unique error. Output from tool will filter out the baseline errors which will at least prevent new errors from being added to CODEOWNERS. As baseline issues are fixed, it's expected they'll be removed from the baseline file. The linter is capable of generating the baseline file as well as using the baseline file to filter out errors.
Known limitations

As previously mentioned, the verification data for team/user, owner org visibility and repo/label data must be generated using a GitHub token with specific permissions on a private pipeline that runs daily. This means that fixing some types of issues will have a delay before their data in Azure storage is updated.
The baseline file means that an existing error can be added to a new line and wouldn't get flagged in CI run. Yes, this is an unfortunate side effect of the baseline file however if this wasn't done, the nightly pipelines running the linter would start out failing and probably never pass. Similarly, if the pipeline runs as part of the PR that has CODEOWNERS changes and there wasn't a baseline then I'll just say hilarity would ensue.

@weshaggard - As per our conversation I'm creating the PR as a draft PR today. The work left to do, outside of any feedback, is mostly around finishing up block formatting error verification, CodeownersFormat.ProcessCodeownersFile tests, baseline file generation and filtering and the README.md for the tool. Though, with this PR description, I've got a good start on the README.

Closing this PR. This one was linter only and the new PR rewrites things which are now Utils that can both parse and lint which will be replacing the existing parser.


Add search and Ballerina pull functionalities

Purpose

Add the functionality to pull a Ballerina module template from Ballerina Central and use it for creating new modules.
Add the search functionality to filter out templates from name in the Ballerina Integrator VS Code extension.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Use leaner alpine:3.1 as Docker base image

The alpine linux distribution is very small and well-suited for Docker images.
$ docker images
REPOSITORY                 TAG       IMAGE ID            CREATED             VIRTUAL SIZE
docker-cleanup-volumes     alpine    ac7a0ef13469        9 seconds ago       24.44 MB
docker-cleanup-volumes     debian    76f6538ac10d        2 minutes ago       234.6 MB

See deis/deis#3716.

Nice! This was something that was bugging me, so much space used for such a tiny script. 24 MB is much better.


Feature: Support for messages on initiative rolls

Add support for a flag like flags.adv-reminder.message.init for showing messages on the initiative roll.

I would love to interact with the initiative roll but the way they ordered things in the system, it's not possible. Advantage Reminder works by using the system-defined hooks and they did things backwards when it comes to initiative. There is a dnd5e.preRollInitiative but it doesn't fire before the dialog for initiative like all the other "pre" hooks do.

There were a few developers that pointed it out and they weren't gonna change the order. I'll gladly reopen this if we get that changed.


Better Music (April Fools)

About the PR

Adds objectively better music to the nuke countdown and space ambience.
(This is a joke.)
Selections include:
Atomic Amnesia MMX
Dies Irae
Rhapsody No. 2
The Blue Danube
Some of the stuff doesn't really line up with the nuke explosion, but this is an april fools PR.
Why / Balance

April Fools!
Does not remove any content, only changes the sound collection.
Technical details

Media

[ ] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Changelog

fix: Made the Nuke and Space Ambience music objectively better. (April Fools!)

i have literally no idea why this is failing 👎

copyright still applies during april fools, probably...

copyright still applies during april fools, probably...

protip, i laid out all the copyright stuff with proper info, use yer eyes

copyright still applies during april fools, probably...

protip, i laid out all the copyright stuff with proper info, use yer eyes

As I read it you actually broke their license. You misattributed it as CC-BY-SA-3.0 but they are actually CC BY-NC-ND 3.0 Deed which specifies you may not make derivatives which you said you've done.

oops, lemme fix that

breach of copyright law has been fixed :trollface:

Never rechecked so 🤪


YAML parser does not support inheritance

There is a YAML feature called "inheritance and include".
Example:
# Define a base configuration
base_config: &base
  key1: value1
  key2: value2

# Inherit from the base configuration
derived_config:
  <<: *base
  key3: value3

The npm yaml package does not support this syntax.
There is an easy fix for this: using js-yaml.
Current error demonstrating that inheritance/include is not working:
 Contents of 'config.yaml' has been parsed as yaml object
Error: Invalid #/applications/app/php (schema path: #/properties/applications/additionalProperties/required) <-- it shows me a required attribute is missing even though it is there in the base
Error: Invalid #/applications/app/<< (schema path: #/properties/applications/additionalProperties/additionalProperties) <-- it interpretes "<<" as property (which is not there)

Hi! Thanks for the note, added js-yaml, updated @master. Can you check it on your examples and confirm it works?

I am testing against the master version, it is working like a charm. Thanks!!

Thanks, merged to v3 as well


update notebooks

Description

Related Issues

Checklist

[ ] Is this code covered by new or existing unit tests or integration tests?
[ ] Did you run make test successfully?
[ ] Do new classes, functions, methods and parameters all have docstrings?
[ ] Were existing docstrings updated, if necessary?
[ ] Was external documentation updated, if necessary?

Additional Notes or Comments

Codecov Report
Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (34830a7) 80.33% compared to head (fff05fe) 67.72%.
Report is 125 commits behind head on main.

Files
Patch %
Lines

superduperdb/backends/base/compute.py
70.37%
8 Missing :warning:

superduperdb/backends/base/metadata.py
66.66%
2 Missing :warning:

superduperdb/backends/base/artifact.py
66.66%
1 Missing :warning:

superduperdb/backends/base/data_backend.py
66.66%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1401       +/-   ##
===========================================
- Coverage   80.33%   67.72%   -12.62%
===========================================
  Files          95      102        +7
  Lines        6602     7234      +632
===========================================
- Hits         5304     4899      -405
- Misses       1298     2335     +1037

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Issue on pydocstyle version 4.0.0

"flake8-docstrings" failed during execution due to "module 'pydocstyle' has no attribute 'tokenize_open'"

Also affected here with incomprehensible message. We are using tox without pydocstyle version pinged.
As workaround, we pinged pydocstyle on tox.ini file to pydcosteyle<4.0.0
Regards!

Looks like the flake8-docstrings people are going to deal with this from their end
https://gitlab.com/pycqa/flake8-docstrings/issues/36

As workaround, we pinged pydocstyle on tox.ini file to pydcosteyle<4.0.0

For those, who use copy-paste to solve issues. Correct version restriction string:
pydocstyle<4.0.0

is it safe to come out yet? I feel kind of badly that pydocstyle is up to version 5.0.1 on pypi now but most of the world has probably pinned to <4.0.0.   Per https://gitlab.com/pycqa/flake8-docstrings/-/issues/36#note_199635329 it should be safe to set flake8-docstrings minimum to 1.3.1 and unpin pydocstyle.

We have yet to experience the aforementioned issue since unpinning and moving to:
flake8-docstrings==1.5.0
pydocstyle==5.0.2

:+1:


[#2769] feat(core): supports schema event for event listener

What changes were proposed in this pull request?
supports schema events for event listener system
Why are the changes needed?
Fix: #2769
Does this PR introduce any user-facing change?
no
How was this patch tested?
existing tests

@jerryshao @yuqi1129 @mchades please help to review when you have time, thanks


Searching not working in overall Application

Searching is not working, for example in categories section when we search some category, it's not working.
Also when I am trying to delete some category, category is not deleting.

The search functionality has been updated to support filtering by both name and slug, so it should now work as expected for categories.
Commit 736cd15
Regarding the issue with deleting categories, the category cannot be deleted because it is still associated with the product table. You need to remove or update the related products before the category can be deleted.


Add instructions in README for lock file version 2

Feature request-
A lot of PRs are changing package-lock.json file mainly because their system has older npm. Add instructions to README such that everybody can check their npm version and upgrade it for lock file version 2 to maintain consistency.
People with lock file version 1 remove some data from package-lock whereas those with version 2 add some data..
Links- https://nodejs.org/en/download/releases/

Note:

Do mention if your are working under GSSoC'21.
If you want to work on an issue, you should check if it has already been assigned to anyone. If the issue is free you can comment /assign to get the issue assigned to you.
If you are raising a new issue and want to work on it then also you should comment /assign under the issue to get it auto assigned.
Please refrain from adding labels to your issue/pull-request on your own. It is the job of the Project Admin and the Mentors to review your issue/pull-request and add labels accordingly.
According to the level of the issue, respective time will be allotted under which the task needs to be completed/resolved by the assigned person. If the issue is not resolved within the given time, it will be assigned to the next person. In case of any discrepancies, the decision of the maintainer will be final.

level - 0: 2 days
level - 1: 3 days
level - 2: 4 days
level - 3: 5 days

Hello, I would like to contribute on this issue under GSSoC'21. Please assign me.

/assign

/assign


PaLM: Error trying to use python API

When trying to use the google-generativeai library in python, I get the following error trying to import:
Traceback (most recent call last):
File "palm_summarizer.py", line 3, in
import google.generativeai as palm
File "/home/bill/.local/lib/python3.8/site-packages/google/generativeai/init.py", line 59, in
from google.generativeai import types
File "/home/bill/.local/lib/python3.8/site-packages/google/generativeai/types/init.py", line 19, in
from google.generativeai.types.text_types import *
File "/home/bill/.local/lib/python3.8/site-packages/google/generativeai/types/text_types.py", line 34, in
class Completion(abc.ABC):
File "/home/bill/.local/lib/python3.8/site-packages/google/generativeai/types/text_types.py", line 49, in Completion
filters: Optional[list[safety_types.ContentFilterDict]]
TypeError: 'type' object is not subscriptable
I'm using Python 3.8; couldn't find a required version in the docs. Does it require a different version? Or is something else wrong here?

Python 3.9 is the minimum supported version, however we accidentally released a package (0.1.0rc2) that required 3.9, but was marked as supporting 3.8. So, if you installed google-generativeai in a 3.8 environment, you'd get the broken package.
I've just yanked 0.1.0rc2 from PyPI, so this should no longer be an issue. 3.8 users will get an old package, but at least now it works.
I'll add a badge to the repo to indicate that 3.9 is the min supported version, too.

I think we've done everything we can here. The bad package was yanked & the Python repo shows the supported Python versions. Feel free to reopen if you think I've missed something.


Remove IDisposable from Hub

It's not needed.

Just for the case, we use it for offloading some work. So hub method may return result to the client immediately and complete work on disposal.

Your specific Hub can be disposable but the base class doesn't need to be.

This is causing issues for me when I have an Observable.Interval subscription on a hub; as the base is checking the dispose and throwing exceptions when the OnNext method of the subscription is fired.  I haven't as yet been able to trace why the hub believes that it has been disposed ?

Not needed anymore


Crosswalk shared mode

Do You plan to support crosswalk shared mode? https://crosswalk-project.org/blog/crosswalk-shared-mode.html

aside from Ionic's plans, note that shared mode is not yet supported in the Crosswalk webview plugin, used by Ionic and Cordova. We are working on it.

Oh... ok - keep up great work!

Shared mode for webview plugin is now supported in Crosswalk 14 and later

It's good news. But I've googling and still no news about that. Can you share us where you get the link? :)

Sorry, it looks like I jumped the gun. We still need to release a new version of the webview plugin before this is usable.


Simplify code by inverting negated ifs and simplifying constructor calls

There were some pieces of code in _ode that were using negated ifs for simple if constructs as in
if not a:
    b()
else:
    c()

Which I inverted to
if a:
    c()
else:
    b()

for simplification and readability. Furthermore, I simplified some if statements. Finally, there was a child class where the constructor was explicitly called where it could have just been a one line call to the super constructor, which I simplified as well.

@perimosocordiae thank you so much for reviewing my PR. I seriously appreciate it!

👍  thanks y'all!

This was merge-ready for a while, so in it goes. Thanks @snood1205, @perimosocordiae


Lifetimes of inner references

We weren't computing inner lifetimes, e.g. Option(&'a ...)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

a test or two might be good since nothing is relying on this (AFAICT)


Failed installation for deepdive runtime deps:Can not run:sudo apt-get update

When i installed deepdive runtime deps,and run this command:sudo apt-get update,i get this error:
"E: Some index files failed to download. They have been ignored, or old ones used instead.",
"W: Failed to fetch http://ubuntu.uhost.hk/dists/trusty-updates/main/i18n/Translation-en  Hash Sum mismatch".
But if i run that update command directly from system,everything is fine.
This error is occured when it uses update command from this command:
bash <(curl -fsSL deepdive.stanford.edu/install) _deepdive_runtime_deps.
Can someone explain for me?

@feiranwang @netj @zhangce I thought we had this all nicely fixed on ubuntu? :)

@astrung Sorry you ran into this.  Could you tell us whether that failing update is happening after some debconf and add-apt-repository commands?  (Creating a gist of your entire output may be much more helpful.)  Have you tried changing your Ubuntu mirror to a more reliable one?  It seems the one you're using is not up-to-date.  I'm not sure how apt-get update could only fail from our installer script because we're not altering anything specially.
@chrismre This runtime dependency installer is actually being tested on Travis upon every build.  However as you know, some unusual user configuration can always break things.  We could take a more aggressive approach to bundle exact versions of all our dependencies to make these errors much less probable.

After i change your Ubuntu mirror,it is fine.Thank you.
But it is neccessary to update system when install Deepdive??

@astrung Glad to hear it worked.  The installer makes sure you have all the dependencies installed upfront to save you from debugging runtime errors.  You can install just DeepDive without any system updates using the deepdive_from_release option.


ngIf with ngAnimate adds ng-enter class to late in Safari 9.0.1

Hi,
I just stumbled across a strange behavior in ngAnimate.
I have build a very simple "collapsible" which removes an element via ng-if and animates its max-height on enter and leave. This works as expected unless I also add an ng-class to a parent which sets the is-open class when the collapsible was opened which also has an animation.
When viewing the plunker in Safari (I have 9.0.1) you will see a very short "flicker". It seems that ng-if
appends the DOM element before the class is set.
Here is a short gif where you can see the problem. When clicking the toggle button it will shortly show the element at full height before collapsing it to max-height: 0 (as per the ng-enter class) and then it starts growing again with the animation.

As soon as you remove the ng-class the flicker problem is resolved. This problem doesn't seem to occur in other browsers (at least not in chrome).
http://plnkr.co/edit/yrebsEs80kPaS90rk0oE

It doesn't seem do be directly connected to ng-class as changing

to

yields the same results.

This problem seems to be related with issue https://github.com/angular/angular.js/issues/12507 in the sense that it applies to the same browsers and was also introduced from 1.4.x versions of angular. I tried your plunker with 1.3.14 and it seems to work fine on Safari 9.0.1.

I can't confirm it, but it does look similar to issue #12507.

I forgot to mention this was on Mac OS X, not iOS.

Sounds also similar to https://github.com/angular/angular.js/issues/12969
@eXaminator can you test this plnkr please? http://plnkr.co/edit/UeScgitwkNM2xqKxL8WO?p=preview
I've added a new style,
.expandable.ng-enter-prepare {
  max-height: 0px;
}

that should make sure the element is hidden before the animation starts.
You can also test with these files directly:
https://rawgit.com/Narretz/angular.js-builds/master/angular.js
https://rawgit.com/Narretz/angular.js-builds/master/angular-animate.js

@eXaminator this is, unfortunately, by design. ngAnimate adds a frame in between the parent ngClass element and the child ngIf. The reason for this is to allow CSS classes to update on the page at the right time in time for the child animations to detect and run properly.
There are two ways to work around this:

Use Narretz's example of using ng-animate-prepare to set pre-styled values which are applied before any of the digests run to trigger animations. Note that this code isn't in the 1.4 branch, but we can add it in here to help with issues like this.

Setup a CSS class to whitelist animations so that traditional, non-ngAnimate invoked, animations (transitions/keyframes) can still operate without ngAnimate interfering.

// app.js
.config(function($animateProvider) {
  $animateProvider.classNameFilter(/use-ng-animate/);
});

      <button ng-click="ctrl.toggle = !ctrl.toggle">Toggle</button>

some content

NgAnimate in Angular 2.0 will be doing a different approach. We will also work to get that behavior into future versions of 1.x.
Closing for now. Please reopen if there is still something that hasn't been covered.

@matsko @Narretz Thanks for all the info on this. I will have a look at this at some point, but we'll probably wait for 1.5 to use the prepare method, which sounds like it is the best approach for this. We're waiting for some of the 1.5 features anyway, hopefully the release won't take too long :)

Awesome, this fixed the issue I was having in safari