Add jQuery like closest feature to tabs component

Thanks @danwhite85. We've been talking more about things like this as part of the NodeList #64 update. We'd obviously like to get away from the DOM traversing that's currently present in a few components. closest and matches polyfills would help a lot with that.
Tooltip, Tabs and Off-Canvas will all require something along the same lines. What are your thoughts on the best way of sharing polyfills between components? Ideally, we're keeping the compiled JS for each as lean as possible, although there will be some unavoidable repetition across components.

On a minor side note, we resolved the e.target issue with nested children for Off-Canvas yesterday #76. Intending to roll the same e.currentTarget fix into Tabs etc. That could be an alternative to the closest amend you've made on line 156?

Nice. e.CurrentTarget is a better solution.
I'm not sure on the best way of sharing polyfills. Perhaps having a polyfills.js distributed with the components that need a polyfill. Obviously there'll some extraneous code if they're just using one component, but it'd avoid duplication if they use multiple.

Cool, thanks for updating that.
We've just rolled the NodeList amends through for PR #82. There might be a minor tweak required to get this branch back on track, sorry.
Also, thinking about the polyfills over the weekend. The matches prefixer in Off-canvas kind of does the trick currently. We could do more of a  native when available and fallback approach? E.g. matches-selector. Still in two minds as whether to augment the native Element.prototype, or play it safe and use a helper util like Tabs closest(). Thoughts?

Playing it safe and using a helper util feels like it's more suited to me. You have a set of isolated components. If the goal is to keep it that way then a helper util makes more sense to me. If in the long-term you plan on having some core.js for all components to share then maybe augmenting the native element makes more sense.

Yeah, definitely in agreement. That way these components can be included in projects, side-effect-free. That sounds good. Shall we branch off again (or merge gh-pages updates into this PR) and try this approach on Tabs? Once we've got a proof of concept, we can roll it through other components that make use of matches and closest.

Hey @danwhite85, we've merged an update into tabs v1.0.5 #87 with closest/matches methods.
As we already had a _closest util in the offcanvas we decided to use that for now, although adding polyfills separately is still something that we need to look into. Thanks for the feedback. 👍


how jobsub_q handles missing -G

if -G is not set, traceback is not expected
old jobsub_q gives a reasonable response if -G is not set.

[rlc@fifeutilgpvm01 ~]$ jobsub_q
Traceback (most recent call last):
  File "/opt/jobsub_lite/bin/jobsub_q", line 114, in <module>
    main()
  File "/opt/jobsub_lite/bin/jobsub_q", line 64, in main
    f"{sys.argv[0]} needs -G group or $GROUP in the environment."
AttributeError: /opt/jobsub_lite/bin/jobsub_q needs -G group or $GROUP in the environment.

This was resolved by making the -G error raise SystemExit  instead of  KeyError; SystemExit does not print a stack trace.


[zh] Update CRICTL_VERSION to v1.28.0

Seems this file has more in-synced content, which you can find by running:
./scripts/lsync.sh content/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md

When I checked it was just that.


fix(v3): fix regression on non-zero plugin exist status

Fixes the regression in helm-3.0.0-beta.5 that swallows and rounds any non-zero exit status greater than 1 from a helm plugin to 1.
This, for example, breaks helm-diff which relies on helm able to return 2 when helm-diff returned 2.
Closese #6788

Although this is now a size/M PR, the body is just addition of 5 lines and removal of 5 lines. Everything else is for tests.
Also - I don't usually write tests for main() but as we've already seen a regression on this feature and I think we've discussed enough the necessity of it at https://github.com/helm/helm/issues/6788#issuecomment-546300454, I believe this should be a good addition to helm's test suite.

cc/ @karuppiah7890

Will review this soon! :)


No colour of face or mtl file?

Hi, how can we use .obj files and not .mrc. I want to have the skin colour of the person also.

Sorry currently this is not the focus of this project :)


add example of pagination links

We are adding a new feature on AMS and to make sure we're getting it right, I think it is important to validate and add examples on json-api.
That is the PR to support pagination links on AMS
What do you thing about that?

@ethanresnick :+1:

I'm pretty sure these need to be complete URIs

@ethanresnick Just to clarify, a relative path is still a "complete URI" correct? i.e. /collection?query

Speficy which strategy on this exmaple as suggested in this note http://jsonapi.org/format/#fetching-pagination:

@dgeb @steveklabnik Thoughts on this example of pagination links?

two small nits, but looks great!

Hi @steveklabnik, before I commit the changes is that okay? Isn't a nice presentation with [and ]?
GET /articles?page%5Bnumber%5D=3&page%5Bsize%5D=1

HTTP/1.1 200 OK
Content-Type: application/vnd.api+json

{
  "data": [
    {
      "type": "articles",
      "id": "3",
      "attributes": {
        "title": "JSON API paints my bikeshed!",
        "body": "The shortest article. Ever.",
        "created": "2015-05-22T14:56:29.000Z",
        "updated": "2015-05-22T14:56:28.000Z"
      }
    }
  ],
  "links": {
    "self": "http://example.com/articles?page%5Bnumber%5D=3&page%5Bsize%5D=1",
    "first": "http://example.com/articles?page%5Bnumber%5D=1&page%5Bsize%5D=1",
    "prev": "http://example.com/articles?page%5Bnumber%5D=2&page%5Bsize%5D=1",
    "next": "http://example.com/articles?page%5Bnumber%5D=4&page%5Bsize%5D=1",
    "last": "http://example.com/articles?page%5Bnumber%5D=13&page%5Bsize%5D=1"
  }
}

I do agree it looks nicer, it's just an invalid URL. :/ What do you think, @dgeb ?

In other examples, we've gone with the unencoded characters for clarity (see the home page example for instance), so @bacarini's being consistent here.
If we want to make a change to encoding all chars, we should do so throughout.

Okay, then I'm fine with not encoding them here.

:+1: all good to merge it?

LGTM - thanks @bacarini :+1:

tks guys!

:100:

@ethanresnick Just to clarify, an absolute path without host is still a "complete URI" correct? i.e. /collection?query

@0x8890 Sorry I'm just responding to this. Revising my original position a bit... I think that providing a partial URI, or, in RFC 3968 parlance, providing a "relative reference" would be allowed but not ideal. That's true whether the relative reference uses an absolute path (as in your example), a relative path, or (as in this original PR) just a query string. Any of those forms should be technically allowed because RFC 3968 provides a mechanism for clients to determine the full URI in these cases. Namely, the client is supposed to use the URI at which the resource was requested as the base URI.
Still, I think a full URI is preferable, in case the document needs to be parsed without knowledge of how it was originally requested; this is the same reason a server might provide a self link. Moreover, unless/until the JSON API spec defines how the self link affects the way a base url is calculated, relying on a relative reference could lead compliant clients to construct an incorrect URI. Consider the following:
GET /articles/1

{
  "data": {
    //...
    "relationships": {
       "comments": {
         "links": {
           "data": [// first 10 resource identifier objects],
           "self": "http://example.com/articles/1/relationships/comments",
           "next": "?page[number]=2&include=comments"
         }
       }
    }
  }
}

Above, the "next" link would be interpreted, per RFC 3986, as having the target URI: http://example.com/articles/1?page[number]=2&include=comments when it "should" be http://example.com/articles/1/relationships/comments?page[number]=2&include=comments. The second interpretation doesn't apply because JSON API doesn't specifically describe the self link in the relationship object as constituting a new, embedded base URI (as would be required, I think, under RFC 3986, section 5.1.1).
So, again, I'd suggest using full URIs whenever possible. However, I think that relative references should be allowed, strictly speaking—though @dgeb, @steveklabnik, or @tkellen may have a different interpretation. Moreover, your comment suggests that JSON API would benefit from a more precise definition of the interplay between self links and RFC 3986 base uris. Would you be open to working up a PR for that?

Changing my position on this again. From the RFC: "A URI is an identifier consisting of a sequence of characters matching the syntax rule named  in Section 3." Therefore, a relative reference is a string that can be converted to a URI, but it's not a URI itself. And, since JSON API says the "self" key et al. must hold a URI, it follows that relative references are simply not allowed. And that's probably a good thing, since extending the spec to allow them would probably break existing clients.

@ethanresnick does that mean that only paths are not allowed anywhere?

Yeah, that's what it would mean. Per my comment in the other issue, though, I'm open to other solutions if you have ideas. I just think that many clients aren't currently expecting relative references (we don't use them in any examples) and would handle them wrong, and I don't think we can blame clientsbfor that, since a relative reference isn't technically compliant, per my logic above

I discussed this with @dgeb way back in the day and we had landed on relative URIs being allowable. We do in fact have examples showing this. There is one shown in the Resource Object section. That said, I'm on the fence about the value of allowing them at this point. Whatever we decide to do, it would be nice to make a non-normative clarification one way or the other.

I discussed this with @dgeb way back in the day and we had landed on relative URIs being allowable. We do in fact have examples showing this.

Ahh, shoot. I totally missed those examples. That complicates things a bit (though the examples aren't normative at least, so taking them out now probably wouldn't be a bc break). Curious what @dgeb thinks.

Whatever we decide to do, it would be nice to make a non-normative clarification one way or the other.

Agreed.
I think that there is some confusion introduced because in many historical documents, the general term for an absolute URI or relative reference that can be converted into an absolute URI, is simply "URI". Take for example RFC 1866 which defines the HREF attribute of an A tag as: "gives the URI of the head anchor of a hyperlink." It goes on to provide examples which clearly indicate relative references.
It seems that in RFC 3986 section 4.1 this concept of a "URI / relative reference" is defined as a "URI-reference".
As @tkellen said, he and I have always intended for relative references to be allowed but not recommended. Given the rather loose ways in which "URI" is used in other RFCs, I think we have some wiggle room to allow for its usage to mean "URI-reference". I am curious to know if there is further evidence of the usage of URI-reference vs. URI in RFCs more recent than 3986.

Given the rather loose ways in which "URI" is used in other RFCs, I think we have some wiggle room to allow for its usage to mean "URI-reference".

I can see that. In fact, even the latest HTML spec seems to use the term "URL" to include what RFC 3986 calls relative references.
If we do (continue to) allow relative references, though, that means we have to reopen #840 and define a procedure for resolving the base URI (which, in an ideal world, we probably don't want to be the same at the document, resource object, and relationship object levels). Who knows how clients have been resolving this so far, so we'd probably need a real-world audit too...

I think it's safe to say there aren't that many implementations in the wild using cross-domain links yet. For example, ember-data works just fine with domain relative "absolute" URIs.

I think it's safe to say there aren't that many implementations in the wild using cross-domain links yet.

Right, I wasn't talking about cross-domain links so much as what path on the current domain to use. E.g.
GET /articles
Host: example.com

{
  "links": {
    // are clients currently resolving this as:
    // http://example.com/?page=2 or
    // http://example.com/articles?page=2
    "next": "?page=2"
  }
  "data": [ /* ... */ ]
}

And, what I was saying about "in an ideal world, we probably don't want [the base url] be the same at the document, resource object, and relationship object levels" was:
GET /articles

{
  "data": [{
    "type": "articles",
    "id": "1",
    "links": {
      "self": "http://example.com/articles/1"
    },
    "relationships": {
      "author": {
        "links": {
          "self": "http://example.com/articles/1/relationships/author",

          // I would want, in an ideal world, for this to resolve to
          // http://example.com/articles/1/author (that is, the self uri
          // in the same link object should form the base uri), but
          // i almost guarantee clients aren't resolving it that way.
          "related": "../author"
        }
      }
    }
  }]
}


feature: support validate volume create config

Signed-off-by: Allen Sun<EMAIL_ADDRESS>1.Describe what this PR did
This PR restricts the validate type Post request config body.

I add VolumeCreateConfig field restriction in swagger.yml;
generate structs in apis/types, in this struct file, there are validation funcs which takes restriction into consideration:

for example the name in VolumeCreateConfig validation:
func (m *VolumeCreateConfig) validateName(formats strfmt.Registry) error {

	if swag.IsZero(m.Name) { // not required
		return nil
	}

	if err := validate.MinLength("Name", "body", string(m.Name), 1); err != nil {
		return err
	}

	if err := validate.MaxLength("Name", "body", string(m.Name), 64); err != nil {
		return err
	}

	if err := validate.Pattern("Name", "body", string(m.Name), `[a-zA-Z0-9][a-zA-Z0-9-_.]{0,63}`); err != nil {
		return err
	}

	return nil
}

When the above two aspects has been done, codes in https://github.com/alibaba/pouch/blob/master/apis/server/volume_bridge.go#L23 will work.
	config := &types.VolumeCreateConfig{}
	// decode request body
	if err := json.NewDecoder(req.Body).Decode(config); err != nil {
		return httputils.NewHTTPError(err, http.StatusBadRequest)
	}
	// validate request body
	if err := config.Validate(strfmt.NewFormats()); err != nil {
		return httputils.NewHTTPError(err, http.StatusBadRequest)
	}

2.Does this pull request fix one issue?
none, this is related to issue https://github.com/alibaba/pouch/issues/443.
3.Describe how you did it
NONE
4.Describe how to verify it
We need to add test cases according to this PR in API POST /volumes/create test.
/cc @Letty5411
5.Special notes for reviews
NONE

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@adb1071). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             master    #452   +/-   ##
========================================
  Coverage          ?   20.3%
========================================
  Files             ?      34
  Lines             ?    1591
  Branches          ?       0
========================================
  Hits              ?     323
  Misses            ?    1233
  Partials          ?      35

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update adb1071...ec91dd0. Read the comment docs.

Moving it to milestone 0.3.


pages/ and server/routes/ handling CommonJS/UMD differently

Environment
N/A, since stackblitz can reproduce it.
Reproduction
https://stackblitz.com/edit/github-ccmqix?file=server%2Froutes%2Fsomeroute.ts
Describe the bug
I use ts-proto to generate grpc-web (and Protobuf) bindings. It generates code that imports @improbable-eng/grpc-web. Node fails to import this:
  import { grpc } from<EMAIL_ADDRESS>  ^^^^
  SyntaxError: Named export 'grpc' not found. The requested module<EMAIL_ADDRESS>is a CommonJS module, which may not support all module.exports as named exports.
  CommonJS modules can always be imported via the default export, for example using:

This goes away if enabling transpilation in nuxt.config.ts. However, with that enabled, now the client side fails (in browser console):
Uncaught SyntaxError: The requested module '/_nuxt/node_modules/@improbable-eng/grpc-web/dist/grpc-web-client.umd.js?v=aa206f89' does not provide an export named 'grpc' (at someproto.ts:1:10)

I've tried a couple of changes to the import (see someproto.ts), but each one causes errors in various ways.
I'm confused about the fact that "server code" seems to have two different flavors: pages running under SSR and what runs from server/. I saw none of this while I was just using pages/ in SSR, without server/.
I suspect they're either using different combinations of CommonJS/UMD import flavors, or there is some difference in TS compilation.
Additional context
The grpc-web module is TS compiled, there is no default export and everything is wrapped in the grpc namespace: https://github.com/improbable-eng/grpc-web/blob/master/client/grpc-web/src/index.ts
Note that the grpc-web package has two compiled files: a main:CommonJS and a browser:UMD (https://github.com/improbable-eng/grpc-web/blob/master/client/grpc-web/webpack.config.js)
Logs
No response

Adding for completeness about grpc-web specifically:
Running a simple ESM file directly in Node v19 results in the same error:
import { grpc } from "@improbable-eng/grpc-web";

However, this is how the grpc-web project itself imports the file in TS: https://github.com/improbable-eng/grpc-web/blob/master/integration_test/ts/src/client.spec.ts
Importing the default works:
import grpc from "@improbable-eng/grpc-web";

While in JS (presumably NodeJS), they use require(...).grpc: https://github.com/improbable-eng/grpc-web/blob/master/integration_test/ts/_proto/improbable/grpcweb/test/test_pb_service.js
They are certainly doing something fishy and relying on TS compilation, but it also seems to be a fairly vanilly TS and webpack project, so I'd expect it to either always work in Nuxt, or never. Right now it depends on where it's imported.

The pages folder is processed by vite; the server directory is only processed by nitro. Both use rollup under the hood, with different configurations.
The root issue is upstream and should be reported to grpc-web - it's invalid Node ESM and can only be consumed by a bundler.
I couldn't reproduce the issue in your stackblitz but would expect something like the below to work:
// https://v3.nuxtjs.org/api/configuration/nuxt.config
export default defineNuxtConfig({
  nitro: {
    externals: {
      inline: ["@improbable-eng/grpc-web"]
    }
  },
})

Thanks for looking into it.

The root issue is upstream and should be reported to

Yepp, makes sense. Funny thing is they had an issue open about it in 2019 but it was claimed to have resolved itself somehow. I've opened a new issue on grpc-web: https://github.com/improbable-eng/grpc-web/issues/1171
Making a webpack ESM library does solve the issue: https://github.com/improbable-eng/grpc-web/compare/master...tommie:grpc-web:esm?expand=1
I would ask if there's an FAQ to add this to, but I guess this issue itself is that now. :)

I couldn't reproduce the issue in your stackblitz

Ah, I've changed someproto.ts back to the bad case. Sorry about that.

but would expect something like the below to work:

Played with inline, and that works as a temporary solution. Thanks!


Arrays with nulls in them result in broken parquet files

Writing a dataframe with an array column when an array contains a null causes hudi to write broken parquet.
To Reproduce
Steps to reproduce (using pyspark here):

Create a spark dataframe where one column is an array, e.g.:

spark_df = spark.createDataFrame([
    (1, '2020/10/29', ['NY123', 'LA456']),
    (2, '2020/10/29', []),
    (3, '2020/10/29', None),
    (4, '2020/10/29', [None]), # this row will break things as would [None,None] or [None, 'ABC']
    (5, '2020/10/29', ['ABC123']),
], ['hudi_key', 'hudi_partition', 'postcodes'])

Write it as hudi with spark

hudi_options = {
    'hoodie.table.type': 'COPY_ON_WRITE',
    'hoodie.table.name': "data",
    'hoodie.datasource.write.recordkey.field': 'hudi_key',
    'hoodie.datasource.write.precombine.field': 'hudi_partition',
    'hoodie.datasource.write.partitionpath.field': 'hudi_partition',
    'hoodie.datasource.write.operation': 'upsert',
    'hoodie.upsert.shuffle.parallelism': 200,
    'hoodie.consistency.check.enabled': True
}

spark_df.write.format("hudi").options(**hudi_options).mode("overwrite").save(hudi_s3_prefix)

Also write the same df as pure parquet for comparison

spark_df.write.parquet(parquet_s3_prefix), 'overwrite')

Hudi uses parquet as underlying format so find the parquet file it wrote on s3 and save a ref to it, same with pure parquet file

Reading the parquet file written with Hudi identified in step 4:

with Spark (spark.read.schema(spark_df.schema).parquet(...)) misses some records (seems to be nondeterministic which ones, could be all of them), e.g. returns only 3 out of 5
with pyarrow (and fastparquet) it fails with an error complaining about the number of values in a column, e.g "ArrowInvalid: Column 7 named postcodes expected length 5 but got length 3" - the number tallies with how many values Spark saw.

Load the data with Hudi

spark.read.format('hudi').schema(spark_df.schema).load(f"{hudi_s3_prefix}/*/*/*") # (globbing for partitions for year,month,day in this example)

dataframe returned will be empty :(

Parquet file written in point 3 (just parquet, no hudi) works fine

Bonus problems
Let's modify the dataframe to no longer contain the [None] for one of the rows
spark_df = spark.createDataFrame([
    (1, '2020/10/29', ['NY123', 'LA456']),
    (2, '2020/10/29', []),
    (3, '2020/10/29', None),
    #(4, '2020/10/29', [None]),
    (5, '2020/10/29', ['ABC123']),
], ['hudi_key', 'hudi_partition', 'postcodes'])

When we repeat the steps above, save with hudi (overriding) to the same location we will be able to correctly read using both spark.read.format('hudi').load(...) and spark.read.parquet(...) but pyarrow will return data with the array column mangled - missing rows, values moved around between rows... e.g.
hudi_key	hudi_partition	postcodes
2	        2020/10/29	    []
5	        2020/10/29	    [None]
1	        2020/10/29	    [ABC123, NY123] # note that ABC123 was a postcode from hudi_key=5 which now has no postcode!
3	        2020/10/29	    None

Expected behavior
It's possible to write a data frame with hudi where one column is an array type and some of those arrays have null(s) in them
Environment Description

AWS EMR version: 5.30.1

Hudi version : 0.5.2 and 0.6.0

Spark version : 2.4.5

Hive version : 2.3.6

Hadoop version : 2.8.5

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : no

Additional context
Override vs append
if we started with some valid hudi data and used mode('append') and attempted to add some more with array containing nulls the new data will not be added but old data will stay and is readable. Later adding "good" (no nulls) data will also work. The parquet file for "bad" batch of data will be broken and can't be correctly read similarly to what was described above.

@umehrot2 @zhedoubushishi any early thoughts?

Small update:
In the post above I described a "bonus problem" where I removed the row with [ null ], wrote it to s3 with hudi,  succeeded to load it back with hudi but failed to load the parquet file with pyarrow. That's still true but I noticed it's the presence of the empty array in one of the rows that is causing this. It may be pyarrow problem, I can't tell but pyarrow will read the parquet file just fine if created with spark without hudi.

Hi @vingov, @umehrot2 , @zhedoubushishi  - did you find anything that might shed some light on the above issue?

I was able to reproduce this issue. The issue indeed seems to be because of the row containing null within the Array. Upon enabling TRACE logging I was able to get the actual exception that was failing the hudi job:
20/12/15 03:47:43 TRACE HoodieSparkSqlWriter$: Error for key: HoodieKey { recordKey=4 partitionPath=2020/10/29}
java.lang.NullPointerException: Array contains a null element at 0
Set parquet.avro.write-old-list-structure=false to turn on support for arrays with null elements.
	at org.apache.parquet.avro.AvroWriteSupport$TwoLevelListWriter.writeCollection(AvroWriteSupport.java:554)
	at org.apache.parquet.avro.AvroWriteSupport$ListWriter.writeList(AvroWriteSupport.java:395)
	at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:355)
	at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:278)
	at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:191)
	at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:165)
	at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:128)
	at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:299)
	at org.apache.hudi.io.storage.HoodieParquetWriter.writeAvroWithMetadata(HoodieParquetWriter.java:83)
	at org.apache.hudi.io.HoodieCreateHandle.write(HoodieCreateHandle.java:116)
	at org.apache.hudi.io.HoodieWriteHandle.write(HoodieWriteHandle.java:159)
	at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:96)
	at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
	at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
	at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Since the job itself does not succeed, Hudi does not find any valid commit at the read time and that is why you see and empty dataframe being returned. Based on the exception above I even tried setting parquet.avro.write-old-list-structure=false in the hadoop config, but after that it fails to write even the first row:
20/12/15 04:04:49 TRACE HoodieSparkSqlWriter$: Error for key: HoodieKey { recordKey=1 partitionPath=2020/10/29}
java.lang.ClassCastException: repeated binary array (UTF8) is not a group
	at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
	at org.apache.parquet.avro.AvroWriteSupport$ThreeLevelListWriter.writeCollection(AvroWriteSupport.java:610)
	at org.apache.parquet.avro.AvroWriteSupport$ListWriter.writeList(AvroWriteSupport.java:395)
	at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:355)
	at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:278)
	at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:191)
	at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:165)
	at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:128)
	at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:299)
	at org.apache.hudi.io.storage.HoodieParquetWriter.writeAvroWithMetadata(HoodieParquetWriter.java:83)
	at org.apache.hudi.io.HoodieCreateHandle.write(HoodieCreateHandle.java:116)
	at org.apache.hudi.io.HoodieWriteHandle.write(HoodieWriteHandle.java:159)
	at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:96)
	at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
	at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
	at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

So, on the surface it seems like parquet-avro support which Hudi uses to write the data in parquet format no longer supports the scenario for Lists containing nulls:
https://github.com/apache/parquet-mr/blob/0a4e3eea991f7588c9c5e056e9d7b32a76eed5da/parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java#L583
Spark uses its own native writers to write parquet which are able to handle this scenario.

@umehrot2
Thanks for looking into this - I'm taking a bit of hope from error message of the code you linked ;)

Have you had any further thoughts on this issue?

@umehrot2
Thanks for looking into this - I'm taking a bit of hope from error message of the code you linked ;)

Have you had any further thoughts on this issue?

@umehrot2 : can you respond to this ticket when you get a chance.

@konradwudkowski : few quick questions as we triage the issue.

Were you running older version of Hudi and encountered this trying to upgrade to a latest version?
Is this affecting your production? trying to gauge the severity.
Or you are trying out a POC ? and this is the first time trying out Hudi.

Hi @nsivabalan I've not tried any previous version, I'm using hudi for the first time now but we're in prod with it. I noticed the problem in a test where this scenario was an edge case I was asserting and was surprised by hudi's behaviour.

So another side effect, when you remove the breaking entry from the example above, hudi_key 4 that has the null in the array:
docker run -d -p 8888:8888 -v ./data:/home/jovyan --name spark jupyter/pyspark-notebook
docker exec -it jupyter/pyspark-notebook /bin/bash
spark-shell   --packages org.apache.hudi:hudi-spark3-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:3.0.1   --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
from pyspark.sql import SparkSession

spark = SparkSession.builder.master('local').getOrCreate()

spark_df = spark.createDataFrame([
	(1, '2020/10/29', ['NY123', 'LA456']),
	(2, '2020/10/29', []),
	(3, '2020/10/29', None),
	(5, '2020/10/29', ['ABC123']),
], ['hudi_key', 'hudi_partition', 'postcodes'])

spark_df.write.parquet("/home/jovyan/spark-out-native.parquet")

spark_df.write.format("org.apache.hudi").mode("append").option("hoodie.table.name", "test").option("hoodie.datasource.write.precombine.field", "hudi_key").option("hoodie.datasource.write.recordkey.field", "hudi_key").save("/home/jovyan/spark-out-hudi.parquet")

quit()

exit
docker cp spark:/home/jovyan/spark-out-native.parquet ./
docker cp spark:/home/jovyan/spark-out-hudi.parquet ./
Then examine the schema, you'll see the spark native write has:
optional group field_id=3 postcodes (List) {
repeated group field_id=4 list {
optional byte_array field_id=5 element (String);
}
}
however, the hudi write has:
optional group field_id=8 postcodes (List) {
repeated byte_array field_id=9 array (String);
}
It appears that hudi writes using the old list format..

Then on the above, if we start pyspark with --conf 'spark.hadoop.parquet.avro.write-old-list-structure=false'
when we write hudi with the same steps as above we get:

spark_df.write.format("org.apache.hudi").mode("append").option("hoodie.table.name", "apple").option("hoodie.datasource.write.precombine.field", "hudi_key").option("hoodie.datasource.write.recordkey.field", "hudi_key").option("spark.hadoop.parquet.avro.write-old-list-structure", "false").option("parquet.avro.write-old-list-structure", "false").option("hoodie.parquet.avro.write-old-list-structure", "false").save("/home/jovyan/apple.parquet")
21/04/30 23:54:57 ERROR HoodieCreateHandle: Error writing record HoodieRecord{key=HoodieKey { recordKey=5 partitionPath=default}, currentLocation='null', newLocation='null'}
java.lang.ClassCastException: repeated binary array (UTF8) is not a group
at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
at org.apache.parquet.avro.AvroWriteSupport$ThreeLevelListWriter.writeCollection(AvroWriteSupport.java:610)
at org.apache.parquet.avro.AvroWriteSupport$ListWriter.writeList(AvroWriteSupport.java:395)
at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:355)
at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:278)
at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:191)
at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:165)
at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:128)
at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:299)
at org.apache.hudi.io.storage.HoodieParquetWriter.writeAvroWithMetadata(HoodieParquetWriter.java:83)
at org.apache.hudi.io.HoodieCreateHandle.write(HoodieCreateHandle.java:118)
at org.apache.hudi.io.HoodieWriteHandle.write(HoodieWriteHandle.java:163)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:96)
at org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
21/04/30 23:54:57 ERROR HoodieCreateHandle: Error writing record HoodieRecord{key=HoodieKey { recordKey=1 partitionPath=default}, currentLocation='null', newLocation='null'}
java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
at org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.startGroup(MessageColumnIO.java:395)
at org.apache.parquet.avro.AvroWriteSupport$ListWriter.writeList(AvroWriteSupport.java:393)
at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:355)
at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:278)
at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:191)
....

So it is looking like even without nulls in an array, we cannot get the 3-level list format? https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules

I guess I know why the config is not honored. Have put up a draft PR here. Would you mind testing it out and letting us know if this works at least for arrays w/ non null and have 3 level list format.
Params that will be honored w/ the fix are:
parquet.avro.add-list-element-records
parquet.avro.write-old-list-structure

@nsivabalan - thanks for the quick fix. I compiled this for spark3 scala2.12 and confirmed it now ran in pyspark without error and produced the new 3-level list format when setting the appropriate params.

awesome, thanks for confirming. Can I close this ticket then?

@nsivabalan - I think the fix allows us to avoid the issues originally reported, but its not the default setting. So people writing lists without the configs may still see the issues shown by @konradwudkowski

got it. We usually don't do any backporting unless there is some regression. This should be available in next release. Closing the ticket for now. Thanks for making hudi better for everyone :) Will add an faq entry that until 0.8.0 3 level list format may not be honored/supported.

ok, thanks. Yes the fix works for me

The fix has been landed and a FAQ has been added here -> https://cwiki.apache.org/confluence/display/HUDI/FAQ?focusedCommentId=181310323#comment-181310323

@nsivabalan With Hudi 0.9.0 version also we are seeing the same issue. Can you confirm if this is fixed.

I also encountered this issue, in my case upgrading to 0.9.0 and setting parquet.avro.write-old-list-structure to false helped both for MOR and COW table.
When using 0.8.0 issue still persists.
In my case I had two problems:

Schema evolution in complex type column from array<struct<col1>> to array<struct<col1, col2>>, col2 should be null, hudi was failing to read table.
While doing upserts, the scenario was like this: do initial load with  array<struct<col1>>, upsert records with exact same data. Hudi was creating avro file with different type for this column: array<string>. It was failing at avro parquet schema conversion, hive sync was failing also because of the schema change.

@kazdy I did a workaround by changing the schema to just string. In any case, the hive table column can be parsed using some UDFs to an array or list

@kazdy could you please share the configurations and environment you used to get this fixed, perhaps thats what we are missing? We are still facing this issue as reported here #5701
Details of our environment description
AWS glue 3.0
Hudi version : 0.10.1
Spark version : 3.1.2
Running on Docker? (yes/no) : no, we are running glue jobs using pyspark

I also encountered this issue, in my case upgrading to 0.9.0 and setting parquet.avro.write-old-list-structure to false helped both for MOR and COW table. When using 0.8.0 issue still persists.
In my case I had two problems:

Schema evolution in complex type column from array<struct<col1>> to array<struct<col1, col2>>, when reading col2 should be null for the first record, hudi was failing to read table.
While doing upserts, the scenario was like this: do initial load with  array<struct<col1>>, upsert records with exact same data. Hudi was creating avro file with different type for this column: array<string>. It was failing at avro parquet schema conversion, hive sync was failing also because of the schema change.

@kazdy can you please share the environment details which you used while fixing this issue, since we are facing still facing it  wanted to confirm if its due to the configurations. Have raised an issue #5701 as well.
Environment Description
AWS glue 3.0
Hudi version : 0.10.1
Spark version : 3.1.2
Running on Docker? (yes/no) : no, we are running glue jobs using pyspark
We even tried downgrading to 0.9.0 but still we got the same error as mentioned in the #5701 .
JARs used:

httpclient-4.5.13.jar,
hudi-spark3-bundle_2.12-0.9.0.jar,
spark-avro_2.12-3.1.1.jar,
libfb303-0.9.3.jar,
calcite-core-1.16.0.jar

@kazdy I did a workaround by changing the schema to just string. In any case, the hive table column can be parsed using some UDFs to an array or list
@stym06 can you please share how you resolved this issue by casting to string. i was following this stackoverflow link to achieve that. But was stuck at requirement of making all incoming records arrays as string.
It seems it works only if we have surrounded the arrays with double quotes\"[]\".
What I tried is to store the inferred schema from array data and store it in another column colSchema, but it didnt help. Im also not sure how to get the data out of column colSchema which holds the schema for the non empty array column.
Schema with all columns cast to string. Any pointers will be highly appreciated:
root
|-- id: string (nullable = true)
|-- some-array: string (nullable = true)
|-- colSchema: string (nullable = true)

Schema post from_json() applied :
root
|-- id: string (nullable = true)
|-- some-array: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- array-field-1: string (nullable = true)
|    |    |-- array-field-2: string (nullable = true)
|-- colSchema: string (nullable = true)

@gtwuser
As far as I remember I was using OSS Spark 3.1.2 with OSS Hudi, I also ran the same checks on EMR  6.4 at the time with the same results.


NEI Subsets / Search issue

When shift clicking a mod from the Subsets dropdown menu, the game inputs the string @Mod.MODNAME into the search bar.
The Mod. is no longer needed and as such returns 0 results for all mods, and instead should just be '@MODNAME'
Also, when viewing the Subsets, the z-index of the menu is incorrect. Items that are in your inventory appear over the subsets menu.

Subsets are still broken.

Shift clicking mod names is now fixed. If there is other issues please open another issue.


Help with Getting Started

I have been trying to get this to work for the past few hours with not much success. I followed the instructions and cloned the repo and ran the start file. Everything seems to work other than some errors which I don't know if they matter. But when I try and get it to use my config that I generated on my windows box and try and use the world I created for the server. Nothing works, it doesn't seem to use my config because experimental mode is still off, the server's name is unchanged, and it resets the world folder with the default files.
What am I doing wrong?

Hey, just a heads up, but that log file contains your steam ID and public IP address, might want to keep those to yourself. I removed the link.

It looks like the ds installs updates, then crashes a couple times before recovering and then starting the game. I'm not 100% sure what's going on but I have a hunch it has to do with the save files you've provided, I don't think you have them in the right location, or there might be some kind of permissions issue. You might try deleting everything, then running the start script, letting the server come up fully (wait for game ready in the log), then stop the server and replace the game files with your own, then start it. Make sure the files are owned by uid 1000. If that doesn't work, try making the permissions on the app data folder more permissive.

Oops, thanks, didn't even notice my ip. I'll try that, I figured it wasn't a perms problem because I thought the scrip was running with sudo perms, because it asks for my password. I wasn't sure where to but the save file because the config folder one wasnt working, so I kinda just put it all over. But I'll try that and see if it works.

closing, maybe the keenswh discord #linux channel could help.


Allow ctr to specify a non-default NATS address

containerd has the  --events-address value, -e flag, but ctr doesn't seem to have a way to subscribe to a non-default address (ex: in ctr run code)
cc @rneugeba

Nats won't last in the final version so its only temporary right now while the core is in development


Debug FAERS Target pipeline

Describe the bug
I have noticed two potential bugs in the output of the FAERS Target pipeline:

The significant target adverse reactions is not exactly a subset of all adverse target reactions.
There are cases where the number of linked targets for a drug is greater than the number of actual targets listing an AE inferred from that same drug.

Observed behaviour

There are significant target/event pairs that do not appear in the unfiltered dataset. Note: This does not occur with drugs AEs.

>>> significantAdverseTargetReactions.select('event', 'chembl_id', 'targetId', 'llr', 'critval').join(
    adverseTargetReactions.withColumnRenamed('reaction_reactionmeddrapt', 'event').select('event', 'chembl_id', 'targetId').distinct(),
    on=['event', 'chembl_id', 'targetId'], how='left_anti'
).show(3)
+-------------------+------------+---------------+------------------+-----------------+
|              event|   chembl_id|       targetId|               llr|          critval|
+-------------------+------------+---------------+------------------+-----------------+
|acute kidney injury|CHEMBL289480|ENSG00000010322|195.84136964049594|8.493369839319712|
|           delirium|CHEMBL289480|ENSG00000010322| 15.25061893917973|8.493369839319712|
|            prurigo|CHEMBL289480|ENSG00000010322|12.377978259030897|8.493369839319712|
+-------------------+------------+---------------+------------------+-----------------+

>>> adverseTargetReactions.filter(F.col('reaction_reactionmeddrapt') == 'acute kidney injury').filter(F.col('targetId') == 'ENSG00000010322').count()
0

CHEMBL3622533 has 14 linked targets. The FAERS analysis only lists 1 target related to CHEMBL3622533.

>>> print(drug_idx.filter(F.col('chembl_id') == 'CHEMBL3622533').show())
+-------------+--------------------+----+
|    chembl_id|             targets|size|
+-------------+--------------------+----+
|CHEMBL3622533|[ENSG00000094631,...|  14|
+-------------+--------------------+----+

>>> print(adverseTargetReactions_2.filter(F.col('chembl_id') == 'CHEMBL3622533').show())
+-------------+-----------------+----+
|    chembl_id|          targets|size|
+-------------+-----------------+----+
|CHEMBL3622533|[ENSG00000051382]|   1|
+-------------+-----------------+----+

Expected behaviour

All significant target/event pairs are included in the unfiltered dataset.
AEs for a target derived from a drug include all the linked targets of that drug.

To Reproduce
The output of the pipeline can be found at: gs://ot-team/mbernal/data/output/fda
A notebook with my analysis is available here: https://github.com/ireneisdoomed/random_notebooks/blob/main/FAERS/OpenFDA_target.ipynb
Additional context
This work is within the context of the extraction of on target significant adverse events from the FAERS data.
The pipeline is a new implementation and essentially mimics the drugs one. Targets are extracted from the mechanism of action dataset. The significant events for a target are then calculated using the Monte Carlo simulation.

Thanks a lot for your review on this @ireneisdoomed ! I must have made a mistake somewhere along the way.
I'll review the openfda computation for targets!
Cheers,
Manuel

@ireneisdoomed I double-checked the issue and it seems the code you pasted below uses drugs as an additional component for the composed index join. Once the target vector is exploded the drug column stops being considered as a groupBy column as the analysis is reaction target pair.


Fix PLDM FRU development code.

Deepak assigned me to fix the FRU table PLDM dev code

Issue is merged


Feature Request: Support markdown files that stored in the repository such as README.md

Now it supports just the URLs in the comment.

This may be dead project


Update ml codeowners

Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:

[ ] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

Given that the SDK has largely been centralized to Nupur's team, can we simplify the ML code paths? It was so split up due to many different teams owning the SDK

Given that the SDK has largely been centralized to Nupur's team, can we simplify the ML code paths? It was so split up due to many different teams owning the SDK

Fair point - will do!


marketing pages missing h1s

h1s are important for both accessibility and seo
every page should have exactly 1 h1

I looked at the layout for the challenge, but I didn't see where the header is so here goes:
Startup in box

https://www.pulumi.com/challenge/startup-in-a-box/

Change "Welcome to Pulumi Challenge" to "Pulumi Challenge: Startup in a box" and make it h1
And remove the other header. All the other challenge pages will follow the same pattern

One Quickstart

https://www.pulumi.com/challenge/one-quickstart/

Change "Welcome to Pulumi Challenge" to "Pulumi Challenge: One Quickstart to Rule them All" and make it h1
Deployments

https://www.pulumi.com/challenge/deployments/

Change "Welcome to Pulumi Challenge" to "Pulumi Challenge: Deployments" and make it h1
Deployments

https://www.pulumi.com/challenge/holiday-shopping/

Change "Welcome to the Holiday Shopping Challenge!" to "Pulumi Challenge: Holiday Shopping" and make it h1
Cloud Engineering
I tried to update to h1, but the font size needs to be fixed for the Cloud engineering days. It should remain the same size as it had with h5. I don't know how to fix it. :(
PR: https://github.com/pulumi/pulumi-hugo/pull/2486


feat: add back helper for generating "parent" path

In v3 of the translate API, we are requiring that folks generate a full parent path when interacting with the API:
parent: `projects/${projectId}/locations/${location}`,

This is fairly unwieldy, and we should reintroduce a helper for this.
CC: @alexander-fenster, @nnegrey

Will be later this week. This will be a compatible change (just adding a few methods). @xiaozhenliu-gg5 is working on it in the generator.

When I look at my project storage bucket in the gcp (or if I query the bucket's metadata from the command line) the value for the location field just says "us (multiple regions in United States)" (or "US")
Am I supposed to assume that it should be "us-central1"?

@arcman77, feel free to open a separate issues for this as that is a little different, but there are different types of cloud storage buckets.
Regions and multi-regions. Sounds like you have a multi-regional bucket.
https://cloud.google.com/storage/docs/locations#available_locations
The translation api currently supports two regions (global and us-central1)
"""
Note: All your resources in a single request to Translation API - Advanced must have the same location. Currently, only global and us-central1 locations are supported. For all custom resources—AutoML models, glossaries, long-running-operations—you must use us-central1. You can also use us-central1 for general translation and language detection. The global location can only be used for general translation models and language detection.
"""
From https://cloud.google.com/translate/docs/migrate-to-v3#resources_projects_and_locations
But the bucket location is only required if you are trying to use custom resources, if you are not, then global should be all you need.


Error message when trying to save structure with Multiple Group type applied to entire structure

Steps to Reproduce

Open Ketcher
Add Benzene on canvas
Select all structure
Press 'S-Group' button and select Type = Miltiple group, Repeat count = 1
Press 'Apply'
Press 'Save' button

Actual behavior
Error message when trying to save structure with Multiple Group type applied to entire structure
Expected behavior
When you press save button, no error message appears and user can save the structure in selected format
Screenshots

Desktop (please complete the following information):

OS: Windows 10
Browser Chrome
Version 112.0.5615.138 (Official Build) (64-bit)

Ketcher version
Version 2.12.0-rc.1

Bug moved to Indigo side https://github.com/epam/Indigo/issues/1168


Revise numbered blocks

In a discussion that was triggered by the implementation of ordered lists, @thoughtstream @lizmat and I came to the conclusion a revision was needed to RakuDoc v2.
The original intent in S26, which was brought into RakuDoc v2, but was missing from previous documentation, was to have numbered lists and headings as extensions of unordered lists and headings. The ordered extension was marked by :numbered or a magical content marker #. This behaviour was not implemented and is confusing, especially because # is currently a comment marker, rather than a number marker.
The resolution is to introduce two new blocks =numitemN and =numheadN, where N >= 1.
A PR will be committed shortly to reflect this change.

@thoughtstream Would you please review the changes.

Thanks, @finanalyst. I've now reviewed the changes.
They are excellent (as usual). I have only a few suggestions (see below).
Note that the final few of these suggested changes do not relate to =numhead or =numitem.
While reviewing the changes to the summary tables, I noted that the :toc, :caption, and :headlevel options
are not available for =graphic blocks. However, graphics like charts and diagrams are often included
in the TOCs of scientific publications (which sometimes even have a separate TOC just for graphics).
So I felt that we should allow these three options on =graphic blocks as well. I am happy to discuss
this further (perhaps in a separate issue), if you wish.
Meanwhile, here is my feedback on the changes...
--- rakudoc_4e4d0b8a6ebebabfe546a5019f3b02dc3d824126.rakudoc	2023-08-28 09:34:23.000000000 +1000
+++ rakudoc_4e4d0b8a6ebebabfe546a5019f3b02dc3d824126_reviewed.rakudoc	2023-08-28 09:37:09.000000000 +1000
@@ -465,7 +465,7 @@
 V<=code> block to be automatically given the same developer note, like so:

 =begin code :allow< B V > :lang<rakudoc>
-    B<<V<=>config item :delta(v2.3.2+, 'oFun enhancements') >>
+    B<<V<=>config code :delta(v2.3.2+, 'oFun enhancements') >>
     =code method droll { say 'what a cool operator' }
     =code method drool { say 'what a slob' }
     =code method drill { say 'keep up' }
@@ -734,7 +734,7 @@
 first four levels of heading, they are required to correctly honour arbitrarily
 nested numberings. That is:
 =begin code :lang<rakudoc>
-    =numhead6 # The Rescue of the Kobayashi Maru
+    =numhead6 The Rescue of the Kobayashi Maru
 =end code

 should produce something like:
@@ -1282,21 +1282,22 @@
 (i.e. as if there had been no intervening non-numbered block).

 The C<:continued> option has no effect on any higher- or lower-numbered C<=numitem> blocks
-that are currently active in the same scope. That is: if C<<A < N < Z>>,
-then the numbering of every active C<=numitemZ> block still resets to 1,
-and the numbering of every active C<=numitemA> block stays the same.
+that are currently active in the same scope. That is:
+a C<=for numitemN :continued> causes
+the numbering of every active C<=numitemZ> block (where I<Z> > I<N>) to reset to 1,
+and the numbering of every active C<=numitemA> block (where I<A> < I<N>) stays the same.

 A C<=numitem> is the same as a C<=numitem1>, by analogy with C<=item> and C<=head>.

-The underlying paradigm is that every sequence of list instructions,
-that is, C<=itemN> or C<=numitemN> instructions,
+The underlying paradigm is that every sequence of list instructions
+(that is: C<=itemN> and/or C<=numitemN> instructions)
 has an I<inherent numeration>. The C<=numitemN> instructions
-express the numeration, while the C<=itemN> instructions do not.
+express the numeration when rendered, while the C<=itemN> instructions do not.

 When a sequence of list instructions is terminated by another sort of block
 then by default a new list is created with its own inherent numeration.
 By specifying C<:continued> on the B<next> list instruction, the previous
-I<inherent numeration> is conserved.
+I<inherent numeration> is preserved and continued.

 =head3 Definition lists

@@ -3123,6 +3124,12 @@
         =cell V<=headN>
         =cell Nth-level heading
     =row
+        =cell V<=numhead>
+        =cell First-level numbered heading
+    =row
+        =cell V<=numheadN>
+        =cell Nth-level numbered heading
+    =row
         =cell V<=defn>
         =cell Definition of a term
     =row
@@ -3132,6 +3139,12 @@
         =cell V<=itemN>
         =cell Nth-level list item
     =row
+        =cell V<=numitem>
+        =cell First-level numbered list item
+    =row
+        =cell V<=numitemN>
+        =cell Nth-level numbered list item
+    =row
         =cell V<=nested>
         =cell Nest block contents within the current context
     =row
@@ -3197,11 +3210,11 @@
             =cell C<=numitem>
         =column
             =for cell :row-span(2)
-            Continue an existing numbering list
+            Continue numbering from the previous numbered list

     =row
         =column
-            =for cell :row-span(6)
+            =for cell :row-span(7)
             C<:toc>

         =column
@@ -3211,9 +3224,10 @@
             =cell C<=input>
             =cell C<=output>
             =cell C<=table>
+            =cell C<=graphic>

         =column
-            =for cell :row-span(6)
+            =for cell :row-span(7)
             Include content or caption in Table of contents

     =row
@@ -3233,7 +3247,7 @@

     =row
         =column
-            =for cell :row-span(8)
+            =for cell :row-span(9)
             C<:headlevel>

         =column
@@ -3245,14 +3259,15 @@
             =cell C<=input>
             =cell C<=output>
             =cell C<=table>
+            =cell C<=graphic>

         =column
-            =for cell :row-span(8)
+            =for cell :row-span(9)
             Define the head level at which the caption is included in the Table of contents

     =row
         =column
-            =for cell :row-span(7)
+            =for cell :row-span(8)
             C<:caption>

         =column
@@ -3263,9 +3278,10 @@
             =cell C<=input>
             =cell C<=output>
             =cell C<=table>
+            =cell C<=graphic>

         =column
-            =for cell :row-span(7)
+            =for cell :row-span(8)
             Caption to be associated with a block in the Table of Contents
     =row
         =cell C<:hidden>

@thoughtstream absence of toc on graphic was an oversight. Thanks for
picking it up
On Mon, 28 Aug 2023, 00:41 thoughtstream, @.***> wrote:

Thanks, @finanalyst https://github.com/finanalyst. I've now reviewed
the changes.
They are excellent (as usual). I have only a few suggestions (see below).
Note that the final few of these suggested changes do not relate to
=numhead or =numitem.
While reviewing the changes to the summary tables, I noted that the :toc,
:caption, and :headlevel options
are not available for =graphic blocks. However, graphics like charts and
diagrams are often included
in the TOCs of scientific publications (which sometimes even have a
separate TOC just for graphics).
So I felt that we should allow these three options on =graphic blocks as
well. I am happy to discuss
this further (perhaps in a separate issue), if you wish.
Meanwhile, here is my feedback on the changes...
--- rakudoc_4e4d0b8a6ebebabfe546a5019f3b02dc3d824126.rakudoc	2023-08-28 09:34:23.000000000 +1000+++ rakudoc_4e4d0b8a6ebebabfe546a5019f3b02dc3d824126_reviewed.rakudoc	2023-08-28 09:37:09.000000000 +1000@@ -465,7 +465,7 @@
V<=code> block to be automatically given the same developer note, like so:
=begin code :allow< B V > :lang-    B<<V<=>config item :delta(v2.3.2+, 'oFun enhancements') >>+    B<<V<=>config code :delta(v2.3.2+, 'oFun enhancements') >>
=code method droll { say 'what a cool operator' }
=code method drool { say 'what a slob' }
=code method drill { say 'keep up' }@@ -734,7 +734,7 @@
first four levels of heading, they are required to correctly honour arbitrarily
nested numberings. That is:
=begin code :lang-    =numhead6 # The Rescue of the Kobayashi Maru+    =numhead6 The Rescue of the Kobayashi Maru
=end code
should produce something like:@@ -1282,21 +1282,22 @@
(i.e. as if there had been no intervening non-numbered block).
The C<:continued> option has no effect on any higher- or lower-numbered C<=numitem> blocks-that are currently active in the same scope. That is: if C<<A < N < Z>>,-then the numbering of every active C<=numitemZ> block still resets to 1,-and the numbering of every active C<=numitemA> block stays the same.+that are currently active in the same scope. That is:+a C<=for numitemN :continued> causes+the numbering of every active C<=numitemZ> block (where I > I) to reset to 1,+and the numbering of every active C<=numitemA> block (where I < I) stays the same.
A C<=numitem> is the same as a C<=numitem1>, by analogy with C<=item> and C<=head>.
-The underlying paradigm is that every sequence of list instructions,-that is, C<=itemN> or C<=numitemN> instructions,+The underlying paradigm is that every sequence of list instructions+(that is: C<=itemN> and/or C<=numitemN> instructions)
has an I. The C<=numitemN> instructions-express the numeration, while the C<=itemN> instructions do not.+express the numeration when rendered, while the C<=itemN> instructions do not.
When a sequence of list instructions is terminated by another sort of block
then by default a new list is created with its own inherent numeration.
By specifying C<:continued> on the B list instruction, the previous-I is conserved.+I is preserved and continued.
=head3 Definition lists
@@ -3123,6 +3124,12 @@
=cell V<=headN>
=cell Nth-level heading
=row+        =cell V<=numhead>+        =cell First-level numbered heading+    =row+        =cell V<=numheadN>+        =cell Nth-level numbered heading+    =row
=cell V<=defn>
=cell Definition of a term
=row@@ -3132,6 +3139,12 @@
=cell V<=itemN>
=cell Nth-level list item
=row+        =cell V<=numitem>+        =cell First-level numbered list item+    =row+        =cell V<=numitemN>+        =cell Nth-level numbered list item+    =row
=cell V<=nested>
=cell Nest block contents within the current context
=row@@ -3197,11 +3210,11 @@
=cell C<=numitem>
=column
=for cell :row-span(2)-            Continue an existing numbering list+            Continue numbering from the previous numbered list
 =row
     =column-            =for cell :row-span(6)+            =for cell :row-span(7)
         C<:toc>

     =column@@ -3211,9 +3224,10 @@
         =cell C<=input>
         =cell C<=output>
         =cell C<=table>+            =cell C<=graphic>

     =column-            =for cell :row-span(6)+            =for cell :row-span(7)
         Include content or caption in Table of contents

 =row@@ -3233,7 +3247,7 @@

 =row
     =column-            =for cell :row-span(8)+            =for cell :row-span(9)
         C<:headlevel>

     =column@@ -3245,14 +3259,15 @@
         =cell C<=input>
         =cell C<=output>
         =cell C<=table>+            =cell C<=graphic>

     =column-            =for cell :row-span(8)+            =for cell :row-span(9)
         Define the head level at which the caption is included in the Table of contents

 =row
     =column-            =for cell :row-span(7)+            =for cell :row-span(8)
         C<:caption>

     =column@@ -3263,9 +3278,10 @@
         =cell C<=input>
         =cell C<=output>
         =cell C<=table>+            =cell C<=graphic>

     =column-            =for cell :row-span(7)+            =for cell :row-span(8)
         Caption to be associated with a block in the Table of Contents
 =row
     =cell C<:hidden>

—
Reply to this email directly, view it on GitHub
https://github.com/Raku/RakuDoc-GAMMA/issues/1#issuecomment-1694794537,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACYZHF3MXZ4ZRPG53JJV3LXXPLMFANCNFSM6AAAAAA4AJBBXY
.
You are receiving this because you were mentioned.Message ID:
@.***>

@thoughtstream Revised text to include =numdefn as discussed in BETA stream.
Please review. Changes start at line 1334.

Thanks, @finanalyst.
I've now reviewed the =numdefn revision.
A few suggestions follow.
(You may also want to check if there actually is a typo on line 743 :-)
--- rakudoc_07ebb5b.rakudoc	2023-08-29 21:32:59.000000000 +1000
+++ rakudoc_07ebb5b_reviewed.rakudoc	2023-08-29 22:02:40.000000000 +1000
@@ -1333,7 +1333,7 @@

 =head4 Numbered definitions

-Definitions can be numbered in the same way as C<=item> lists, that is a C<=numdefn> instruction
+Definitions can be numbered in the same way as C<=item> lists. That is, a C<=numdefn> instruction
 will express the I<inherent numeration> of the definition list when it is rendered.

 As with ordinary lists, if a C<=numdefn> instruction is associated with a C<:continued> metadata option,
@@ -1356,7 +1356,7 @@
     finals
 =end code

-will yield something like
+will yield something like:
 =begin code :allow< B > :lang<RakuDoc>
 1. B<AAA>
   first letters
@@ -1369,7 +1369,7 @@
   finals
 =end code
 In order to continue a list, an explicit C<:continued> is needed, such as:
-=begin code :lang<RakuDoc>
+=begin code :allow< B > :lang<RakuDoc>
     =numdefn AAA
     first letters

@@ -1378,7 +1378,7 @@

     A normal para

-    =for numdefn :continued
+    =for numdefn B<:continued>
     XXX
     ending letters

@@ -1399,9 +1399,11 @@
   finals
 =end code

-Suppose a writer wants to have a single continuous list of definitions within the same
-block scope (for example a section or for the whole document), then every C<=numdefn>
-could be associated with a C<:continued> as follows:
+Suppose a writer needs every numbered definition to form a single monotonic sequence
+within some block scope (e.g. for every definition within a document section,
+or every definition throughout the entire document). This would require every C<=numdefn>
+to be specified with a C<:continued> option. Or the writer could simply pre-configure
+every C<=numdefn> within the scope, as follows:
 =begin code :lang<RakuDoc>
     =config numdefn :continued
 =end code
@@ -2899,7 +2901,7 @@
 =begin code :lang<rakudoc>
 The emergency signal is:
 dot dot dot   dash dash dash    dot dot dot.
-=end codetypo line 743
+=end code
 rather than:
 =begin code :lang<rakudoc>
 The emergency signal is: dot dot dot dash dash dash dot dot dot.
@@ -2928,7 +2930,7 @@

 Which produces:

-=nested Raku is multi-paradigmatic N<Supporting Procetypo line 743dural, Object Oriented, and Functional programming>
+=nested Raku is multi-paradigmatic N<Supporting Procedural, Object Oriented, and Functional programming>

 =head3 Keyboard input

@@ -3051,7 +3053,7 @@
 =config C :allow< B I U >
 =config V :allow< N L X >
 =end code
-typo line 743
+
 =head3 Indexing terms

 Anything enclosed in an C<X<>> code is an B<index entry>. The contents
@@ -3119,7 +3121,7 @@
 For example, suppose a Rakudoc document is used to create part of a website in which a button is to be styled
 that accesses the API of a payment service (e.g. 'PayMe'). A developer can then create some functionality according
 to the rules of a RakuDoc renderer, and present it to the service point. In an HTML document, this might be coded
-as a C< <button class="PayMeApp" data-token="vendor-idtypo line 743" data-price="$0.99"> > with a specific class and structure.
+as a C< <button class="PayMeApp" data-token="vendor-id" data-price="$0.99"> > with a specific class and structure.
 The RakuDoc instruction might then be:
 =begin code :lang<Raku> :allow<B>
     To sample this marvelous libation B<M<order now by PayMeApp|PayMeApp; vendor-id; $0.99>>
@@ -3452,7 +3454,7 @@
         =cell Markup instruction
         =cell Specifies
     =row
-        =cell V<A<...>>typo line 743
+        =cell V<A<...>>
         =cell Alias to be replaced by contents of specified V<=alias> directive
     =row
         =cell V<B<...>>

There was once a typo on 743! (Sounds like the start to a modern Brothers Grimm) How the phrase got into the copy bin and multiply pasted, I don't know. But I did have a problem a couple of days ago pushing to github without a valid security token. Hopefully, its all sorted.

With the patch, I think this issue is resolved.


[Feature request] Postgresql : edit roles properties

Hi,
Except if I missed something, for the moment, roles informations in DBeaver are not editable.
It would be nice if we could edit properties, modify values, which would produce and execute the "ALTER ROLE..." SQL command corresponding to what has changed.
Thanks in advance.
Regards,
Cedric

Related to this post, I created a new login role but found no way to add the login role to a group role. I added the login role to the group role using a sql command. After doing that, I could see the role was added in the properties of the group role but the Role (name) column was empty for the new role until closing out of DBeaver and opening it again.

Is there any plan on this Feature request ?

I see the role I created earlier, but I cannot set a password for it. There's no UI for that. Need to look up the SQL command for that.

will be implemented in the scope of #8133


Investigate PySpice support

Pyspice seems kinda hard to use. I will shelve this for now until someone has a genuine use case for it


Reformat the code

Currently the code has different formatting in different places. Once some important PRs are merged, it makes sense to reformat the whole crate with rustfmt, according to selected style.
Basically running cargo fmt will improve the situation a lot.

@waywardmonkeys can you please run the cargo fmt right after everything merged before 0.5 release? So the code would be a bit better for a new release.


Add buffered output plugins for tests

Revised: #518.
And I've added tests for added buffered output plugins for testing.
Remaining Tasks

[ ] Rewrite new TestDriver and new v0.14 API (Not Yet)

@cosmo0920 Can you leave this pull-request as is for a while?
I think we can add these only for v0.14, and that version will have another (well-organized) test drivers.

Yup, I would like to leave as is for now.

I'm sorry but I implemented out_buffered_output in #972 because it's very good for a testbed of test drivers.

Hmm, a build on AppVeyor failed, but there are no details.

Thank you!


Visual Studio 2019 Support

Please add support for Visual Studio 2019  https://marketplace.visualstudio.com/items?itemName=ChrisDahlberg.StyleCop

Still cant install VSIX on VS2019 :/


Add some unit test cases for garray

Codecov Report

Merging #235 into master will increase coverage by 1.52%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #235      +/-   ##
==========================================
+ Coverage   75.95%   77.48%   +1.52%
==========================================
  Files         207      207
  Lines       17834    17834
==========================================
+ Hits        13546    13818     +272
+ Misses       3629     3399     -230
+ Partials      659      617      -42

Impacted Files
Coverage Δ

g/os/gspath/gspath_cache.go
81.81% <0%> (-4.55%)
:arrow_down:

g/container/gtree/gtree_avltree.go
97.5% <0%> (-1%)
:arrow_down:

g/container/gtree/gtree_btree.go
91.87% <0%> (-0.68%)
:arrow_down:

g/container/gtree/gtree_redblacktree.go
96.95% <0%> (-0.41%)
:arrow_down:

g/container/garray/garray_normal_int.go
98.88% <0%> (+10.83%)
:arrow_up:

g/container/garray/garray_sorted_int.go
99.36% <0%> (+14.05%)
:arrow_up:

g/container/garray/garray_normal_string.go
99.44% <0%> (+14.44%)
:arrow_up:

g/container/garray/garray_normal_interface.go
100% <0%> (+14.73%)
:arrow_up:

g/container/garray/garray_sorted_interface.go
99.36% <0%> (+15.28%)
:arrow_up:

g/container/garray/garray_sorted_string.go
99.35% <0%> (+15.48%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7d30dd...5da822f. Read the comment docs.


Update Cosmos SDK

Update to Azure.Cosmos from Microsoft.Azure.Cosmos
References #66

Closing as dupe of #78


[REQUEST]

Here is what i want
bool passwordtest =<EMAIL_ADDRESS>@"[^a-zA-Z0-9~!@#$%^&*()_+-={}[]|\:;<>,.?/]");
IF BOOLKEY @passwordtest Is true
data.STATUS = "FAIL";
return;
END
a simple ASCII checker for input.DATA which marks the data as fail with a check box or something

You can simply go to the Config > Settings and use a Regex data rule to check this. The feature is already available.


[WiP] Allow to apply recursive function to directory

Hello,
I'm trying to use your library for recursive treatments, where Callback also takes as an argument the return value of Callback(child elements). This would allow to code things like 'du' (print directory  size - including all descendants -, for every directory encountered). That is, to have callback functions which are of type like :
func(osPathname string, directoryEntry *Dirent, sibling, child RecurseResult) (RecurseResult, error)
where RecurseResult is any type (but same for result and params of the callback), 'sibling' is the result of the callback on previously treated elements in the same directory and 'child' is the result of the callback on children (more exactly, on the last children treated).
I'm a beginner go programmer, so I may have made errors or made misconceptions. How i did  :

i copied code from walk.go into recurse.go
i switched PostChildrenCallback to PreChildrenCallback (as 'Callback' in 'recurse' will always be called after children).
changed some code order in the main 'walk' (now 'recurse') function
i added RecurseResult and RecurseFunc types.

The problem is that i don't know how to make go accept any type as 'RecurseResult'. When I try to define RecurseResult as 'interface {}' :

./main.go:16:11: cannot use func literal (type func(string, *godirwalk.Dirent, int64, int64) (int64, error)) as type godirwalk.RecurseFunc in field value

If I manually switch from RecurseResult to int64 type in the code, it works, but ... well ... that's not the goal.
Do you have any idea ?
Are you interested in this 'recursive thing' ?
Note: if it works, the classic 'walk' function could be considered as a particular case of recursive function (a one which doesn't use the result from children and doesn't return any particular value), except the difference in order of treatment (children before or after parent).

Just some quick thoughts after a cursory overview of your code and proposal (I'm not karrick, btw).  It's a lot of code duplication, and it's not clear that it would be trivial to make Walk() a wrapper around your Recurse() func.  For one thing, the Callback signatures are different, and would require munging.  And as you noticed, it gets tricky when deciding what type is used to pass information about siblings and children to the Callbacks.  Making it an interface{} (or your RecurseResult type) means the users will likely have to deal with type assertions, etc.  So it seems non-trivial.
Think about how you'd build this "du" program if you absolutely had to use the existing Walk() func.  Well, the only identifying information that the current callbacks get is a filename, which can identify the current dir entry, and it's parent dirs (using path.Dir()).  This can be used to create and fill a map (declared outside the callback funcs but accessible to them via closure), where each key is a directory path, and as each filesize is calculated, it adds to the dir's amount.  As the end of the walk, you'd have the size count for each directory (but not it's children).  You could then do a post-processing pass to find the cumulative counts.  Or alternately, you could use the existing PostChildrenCallback to do it in place; when a dir is finished, you find the parent of that dir in the table and add the dir's sum to it.
Basically, I'm saying that there are ways to do what you want given the existing mechanism, which involve using an outside datastructure available to the callbacks, rather than trying to pass generalized data around via the empty interface.  The benefit is that it keeps the walk code simpler, and it probably makes the user's work easier than trying to work with the Recurse() func.
It is true that having to do a path.Dir() on each Callback seems a bit wasteful when the Walk() function has that information; it would be nice if that had been one of the arguments to the Callbacks.  Perhaps there could be another set of callbacks that provides additional information known to the Walk() func on each call, if the right information/interface for the callbacks could be determined.  Some other walk implementations will (for example) have a callback for each new dir encountered, and return a list of file and subdirectory entries for each visited dir, which might make some problems (like your 'du' idea) a bit more straightforward to implement, though the performance and memory implications of building the arbitrarily large lists need to be considered.

Thanks for the constructive comment !

And as you noticed, it gets tricky when deciding what type is used to pass information about siblings and children to the Callbacks. Making it an interface{} (or your RecurseResult type) means the users will likely have to deal with type assertions, etc. So it seems non-trivial.

Well, I thought that defining the RecurseResult type (rather than using directly interface {} type) would imply that the RecurseFunc uses the same type for return value and for children/sibling parameter. Isn't it the case ? (anyway, for now it doesn't work at all...)

Think about how you'd build this "du" program if you absolutely had to use the existing Walk() func

That's what I wanted to do for telegraf PR 4778. I thought about your first solution (big table of all counts before re-parsing the tree to sum it all), but it could take a lot of memory (think about running it on '/home', which is basically my goal), that's why I didn't try.
Your second solution (updating the parent's size with PostChildrenCallback) seems a better solution to me, as it's also possible in PostChildrenCallback to delete unnecessary information (current dir's size) as soon as it has been used (printed out or anything else). This way, the memory footprint is O(folder depth) rather than O(total number of files) I think.

This way, the memory footprint is O(folder depth) rather than O(total number of files) I think.

I think, for what I was describing, the worst case memory footprint would be O(total number of directories), which hopefully is generally about an order of magnitude less than files; but yes, if it's feasible, then culling the map after each directory is complete seems like the most elegant way to do it, and hopefully somewhat straightforward.
FWIW, I was able to get your du-fast version to compile using the RecurseResult type, with some simple changes; Replacing the int64 in the Callback signature to RecurseResult, and then using type assertions inside the callback on the siblingSize and childSize vars, so that they can be added as int64s.  This is on the earlier edc343a commit, before you changes the Recurse func to use int64.  Just in case you want to explore that some more; it can be illustrative.  Needing to use type-assertions in the callback is (imo) a serious drawback to that approach.  Although it provides potential power, it adds a burden to callback implementors.  Anyway, if you want I can supply a diff, but it's probably worth you trying it first (the compiler feedback should get you most of the way there).

Is there a reason that the value could not be a closed over variable?
package main

import (
    "fmt"
    "os"
    "path/filepath"

    "github.com/karrick/godirwalk"
)

var progname = filepath.Base(os.Args[0])

func main() {
    dirname := "."
    if len(os.Args) > 1 {
        dirname = os.Args[1]
    }

    var size int64

    err := godirwalk.Walk(dirname, &godirwalk.Options{
        Callback: func(osPathname string, de *godirwalk.Dirent) error {
            if !de.IsRegular() {
                return nil
            }

            st, err := os.Stat(osPathname)
            if err != nil {
                return err
            }

            size += st.Size()
            return nil
        },
        ErrorCallback: func(osPathname string, err error) godirwalk.ErrorAction {
            // Your program may want to log the error somehow.
            fmt.Fprintf(os.Stderr, "%s: %s\n", progname, err)

            // For the purposes of this example, a simple SkipNode will suffice,
            // although in reality perhaps additional logic might be called for.
            return godirwalk.SkipNode
        },
        Unsorted: true, // set true for faster yet non-deterministic enumeration (see godoc)
    })

    if err != nil {
        fmt.Fprintf(os.Stderr, "%s: %s\n", progname, err)
        os.Exit(1)
    }

    fmt.Println(size)
}

I believe that using closures is a more general solution to this sort of problem. Please re-open if you feel that I have not completely understood your use-case.

Thanks for examining this PR.
I tried to get the total size for the main folder and all its descendants to get a result like 'my_dir/ 50000; my_dir/a/ 25000; my_dir/a/1/ 20000; my_dir/a/2/ 5000; my_dir/b/ 10000; my_dir/c/ 15000'. If I'm not wrong, the only thing you get with your code is 'my_dir/ 50000'.
But anyway, using the PostChildrenCallback like proposed @chadnetzer, it's possible without modifying godirwalk. I implemented it in telegraf/filecount (where we count both size and number of files under each directory).
I find this solution a bit less elegant, as it implies maintaining the result array "by hand", while the solution I proposed - if it worked - would have made possible to hide these arrays in the call stack (more in a "functional" way of coding), but it has the merit to work, while mine doesn't !

Ah, I see your point better I think. I went ahead and coded up a more complete example that shows how to effect the desired output, and added it to the examples/sizes directory:
https://github.com/karrick/godirwalk/blob/master/examples/sizes/main.go

Here's the example:
package main // imnport "github.com/karrick/godirwalk/examples/sizes"

import (
    "fmt"
    "os"
    "path/filepath"

    "github.com/karrick/godirwalk"
)

var progname = filepath.Base(os.Args[0])

func main() {
    if len(os.Args) < 2 {
        fmt.Fprintf(os.Stderr, "usage: %s dir1 [dir2 [dir3...]]\n", progname)
        os.Exit(2)
    }

    scratchBuffer := make([]byte, 64*1024) // allocate once and re-use each time

    for _, arg := range os.Args[1:] {
        if err := sizes(arg, scratchBuffer); err != nil {
            fmt.Fprintf(os.Stderr, "%s: %s\n", progname, err)
        }
    }
}

func sizes(osDirname string, scratchBuffer []byte) error {
    var sizes []int64 // LIFO (push, pop semantics)

    return godirwalk.Walk(osDirname, &godirwalk.Options{
        ScratchBuffer: scratchBuffer,
        Callback: func(osPathname string, de *godirwalk.Dirent) error {
            if de.IsDir() {
                sizes = append(sizes, 0)
                return nil
            }

            st, err := os.Stat(osPathname)
            if err != nil {
                return err
            }

            size := st.Size()
            sizes[len(sizes)-1] += size

            _, err = fmt.Printf("%s % 12d %s\n", st.Mode(), size, osPathname)
            return err
        },
        PostChildrenCallback: func(osPathname string, de *godirwalk.Dirent) error {
            i := len(sizes) - 1
            var size int64
            size, sizes = sizes[i], sizes[:i]

            st, err := os.Stat(osPathname)

            switch err {
            case nil:
                _, err = fmt.Printf("%s % 12d %s\n", st.Mode(), size, osPathname)
            default:
                // ignore the error and just show the mode type
                _, err = fmt.Printf("%s % 12d %s\n", de.ModeType(), size, osPathname)
            }
            return err
        },
    })
}

I tested it : that's what i looked for. Thanks.
Clever use of the LIFO stack to compute the total size !

Although it’s not always obvious at first glance how to take advantage of it, and it can be severely abused to the detriment of code clarity, a combination of closures and callback functions do allow a tremendous amount of power.


feat: ArgoCD support

For every application, myks generates an ArgoCD Application resource with minimal viable configuration which can be changed via data values and overlays (in _env/argocd and _app/*/argocd).
For every environment, myks generates an ArgoCD AppProject and Secret resources. The resulting resources can be changed via data values and overlays (in _env/argocd and _app/*/argocd).
It is possible to set custom names and prefixes to the resources from data values files. The rest for now should be set via overlays.

LGTM.
nit: your PR summary is using _app/*/argocd instead of _apps/*/argocd ;)


Update to use add_compile_definitions instead of C_FLAGS

Discussion here on another PR:
https://github.com/Azure/azure-sdk-for-c/pull/763#discussion_r424842003
We should try to avoid mucking with C_FLAGS if we can and use the CMake primitives to modify the compilation.

Closing since my man @vhvb1989 already did it in the other PR.


Can't change settings after initial configuration

Hi there. We're having some challenges using this image since once the data volumes are created and configured, there is no way to modify the configs with future updates.
Say, for example, we want to add a new forward server, app, etc. I could update the ENV variables to accomplish this when first starting a container. But if the data volumes already exist, I have to either completely delete them or log into the container and run commands manually.
It might be nice to have an ENV variable to run splunk commands even if the server is already configured? Or perhaps a flag to re-run configuration regardless of what it finds in the volume (assuming that Splunk properly handles the case of identical commands being re-run)?
Thanks!

@lsglick this is by design.
Intention is pretty simple, when this is a fresh instance - we can safely call these commands, if not - there are can be a lot of undefined situations. Let's say that we will implement a flag "RERUN_ALL_COMMANDS" - what will happen when you server will be restarted and restart policy for this container is set to always restart? These commands will be invoked again and again.
My original Intention for these environment variables was to have them for stateless forwarders, for which you don't really have to keep var and etc folders, you can safely start them clean with using these environment variables.
If you want to modify splunk configuration after upgrade there are two recommended ways of doing it
Call splunk binary
docker exec -it container_name entrypoint.sh splunk status

Or open bash in container and invoke several commands
docker exec -it container_name entrypoint.sh splunk-bash

(big thank to @bshuler for https://github.com/outcoldman/docker-splunk/pull/19)

Thanks for the explanation! We've been using your suggested way, I was just hoping for a more seamless way.
I'll probably fork and add a local set of ENV variables for commands to always run regardless of if configured. You're right our instance isn't stateless since the scripted_input keeps track of its progress uploading data.
Specifically I want to execute something like: install app /tmp/my_app.tar -update 1 -auth admin:changeme so it shouldn't hurt anything to run it extra, but will ensure if I update my_app, the latest version is installed on the volumes.


LDAPJS keeps trying to connect to LDAP when no bind attempt occurs

I have taken multiple Packet Captures trying to figure out why ECONNRESET's occur.
I have found that even though you are not attempting to do any binds or LDAP actions that LDAPJS will try to connect to our LDAP server and eventually disconnect and just keep trying to re-connect.
This behavior doesn't seem to be by design. Can we get some clarification on this?

I was fighting with this recently and after reviewing the client file in the library I realized the idle event wasn't being emitted if I didn't create a bind first. I haven't spent too much time reviewing this code to see what is going on, but if I have time one day maybe I can create a pull request. For now I will likely default back to using OpenLDAP library.
You can hook up to the idle event like so:
const client = ldap.createClient({
    url: `ldap://my-dc`,
    idleTimeout: 10 // Close due to inactivity
});

client.on('idle', event => {
    console.log("Timed out")
});

// Time out is never printed

The result of this was I never received a idle event, so I decided to put a test bind in there and I started to received the idle event.
const ldap = require('ldapjs');
const client = ldap.createClient({
    url: `ldap://my-dc`,
    idleTimeout: 10 // Close due to inactivity
});

client.bind("user@domain.com","Password", (err,res) => {
    if(err)
        console.log(err)
});

client.on('idle', event => {
    console.log("Timed out")
});

// Console now prints Time Out

👋
On February 22, 2023, we released version 3 of this library. As a result, we are closing this issue/pull request.
Please see issue #839 for more information, including how to proceed if you feel this closure is in error.


Permissions

At a minimum can bootstrap off the existing create-table and insert-rows permissions in Datasette 1.0.

I'm going to use a custom datasette-extract permission for this - it costs money to run the GPT-4 API so having a separate permission makes sense.

I'm going to require both create-table and insert-rows as well, otherwise installing this plugin might accidentally give users a backdoor way to create tables that they did not have before.

These need to be taken into account by the code that shows the menu items too.


How did you manage to handle this large project from scratch?

Gitea is community fork of Gogs so it was not really created from scratch by us but anyway it takes a lot of time and dedication to keep developing it


Allow configurable response for non-SNI requests

Do you want to request a feature or report a bug?
Feature
What did you do?
Traefik is responding with its certificate 'TRAEFIK DEFAULT CERT' for non-SNI aware clients,
please provide a configurable option per frontend to set a domainname
certificate to return.
eg.
When a non-SNI aware client requests TLS secured traffic from www.domain.name,
consult something like the below imagined option to respond with the certificate for www.domain.name, rather than the 'TRAEFIK DEFAULT CERT' :
traefik.frontend.xyz.nonSNICertDomain=www.domain.name

hi all ,
i also have currently a issues with a customer which uses tools which doesn't seem to support SNI.
There where complaining about bad "TRAEFIK" certificates which was not allowed by there corporate security policy. The fact that we present a self signed certificate is maybe not so good response.
i would like to see that clients which do not support a valid SNI connection, we log a error about this client (ip address) and drop the connection completely. This way we see this kind of connections error quickly en inform our customer that they need to to use more "modern" tooling /libraries .
my point of view is that if you don't support SNI , which is critical to have many site hosted using single ip , you beter upgrade your tooling or just leave and go fishing. SNI is so old (introduced in RFC3546 , in 2003 ) . if the client doesn't support this , he's better of to not go on the internet , because how well will his stack be protected against all issues currently known .
so something like "drop non SNI client " feature would greatly help this.
something simular like apache has (http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslstrictsnivhostcheck)


When building and publishing as a web application, use the webviewx plug-in to ask questions

The current page uses sliverpersistentheader to hover and WebView together. Click the component on the hover interface, and it will get stuck

Using stack, add WebView and (flutter native components with click events, such as draggable and gesturedetector), and click events will be transmitted to WebView

Hi @yudanh,  WebviewX is not a first part plugin (developed by flutter team) issues related to that must be filed with the maintainer closing as this is not a bug in flutter


Adding container-lifecycle-hooks in Vietnamese

Signed-off-by: Nguyen Hai Truong<EMAIL_ADDRESS>

/lgtm
/approve
Thanks to @truongnh1992


The reconstructed image using codec.py is inconsistent with the original image

Thank you for your contribution!
I used one computer to compress the original image into a code stream, and then used another computer to decode it [the hardware configurations (CPU and GPU) of the two computers are inconsistent], and the reconstructed image has many black square dots. If you use the same computer for encoding and decoding, there will be no such problem. If the configurations of the two computers are exactly the same, this problem will not occur. Is this normal?

Corrupted reconstruction of images often occurs because the decoder decodes a different latent than what was encoded. There may be other possible reasons (e.g., the input image lies far outside the dataset of distributions that the model was trained on), but let's focus on the most common one.
Often, the encoder and decoder are not "in-sync". Side-information and autoregressive models are the most susceptible to this. This is because even small differences in the encoding distribution (e.g., discretized Gaussian) at the decoder-side can result in incorrectly decoded symbols. This in turn throws the context model (which relies upon previously decoded symbols to generate new encoding distributions) out-of-sync. This results in a cascading series of accumulating errors, and everything blows up quite quickly.
Using a precomputed discretized Gaussian table (which CompressAI models usually do) offers a bit of built-in protection against differences in reconstructed scales ($\sigma$). The protection for decoding a given symbol is probabilistically proportional to the density of the floating point values within the operating range, though it is not perfect, either. The means ($\mu$) might be less happy here, though, since they are not quantized, and are very likely to vary.
What causes these small differences in computed values, which then throw the decoder out-of-sync? A common reason is the use of non-deterministic operations. Even the same sum/reduction operations executed on the same hardware might give different results across different runs. See this example. (e.g., consider: (0.1 + 0.2) + 0.3   !=   0.1 + (0.2 + 0.3).)
Some proposed remedies usually involve quantization of all the operands involved in the operations. Though even here, one must be careful to ensure properties like associativity (a + b) + c = a + (b + c) hold to facilitate provable determinism. There's probably more about these remedies in recent research papers, if you're curious.

Related issues/comments:

https://github.com/InterDigitalInc/CompressAI/issues/235#issuecomment-1740940306
https://github.com/InterDigitalInc/CompressAI/issues/229#issuecomment-1526259843
https://github.com/InterDigitalInc/CompressAI/issues/54#issuecomment-843452964
https://github.com/InterDigitalInc/CompressAI/issues/47#issuecomment-831652510

Thank you very much for your reply, I will check the possible reasons.

由于解码器解码的潜在信息与编码的潜在信息不同，因此经常会发生图像重建损坏的情况。可能还有其他可能的原因（例如，输入图像远远超出模型训练的分布），但让我们关注最常见的一个。
经常发生的情况是编码器和解码器不“同步”。边信息和自回归模型最容易受到此影响。这是因为，即使解码器侧的编码分布（例如，离散高斯分布）中的微小差异也可能导致符号解码不正确。这反过来又导致上下文模型（依赖于先前解码的符号来生成新的编码分布）不同步。这会导致一系列级联的累积错误，一切都会很快崩溃。
使用预先计算的离散高斯表提供了一些内置的保护，以防止重建尺度的差异（σ）。compressCompressAI 模型通常在/期间使用此功能decompress。对单个符号进行解码的保护在概率上与操作范围内的浮点值的密度成正比（即，对于float32 ，p ≈ 1e-6 ...尽管它可能更好，具体取决于情况...）。不幸的是，手段（μ）不过，这里可能不太高兴，因为它们_没有_量化，并且在不同的运行中可能会有所不同。
是什么导致了计算值的这些微小差异，从而导致解码器不同步？一个常见的原因是使用非确定性操作。即使在相同硬件上_执行相同的求和/归约操作，在不同的运行中也可能会产生不同的结果。_

非确定性归约
此外，由于硬件计算的不确定性，当前的实现可能不会在不同设备上产生相同的结果。因此，设备 A 上的编码和设备 B 上的解码可能不起作用。即使是同一个设备，如果不能确定地进行矢量浮点计算，也可能会失败。例如，归约和的结果取决于归约顺序，归约顺序可能会根据首先完成的计算单元、硬件单元强制执行的归约树或其他原因而变化。
(0.1 + 0.2) + 0.3   !=   0.1 + (0.2 + 0.3)

评估每一方都会给出不同的结果：
>>> (0.1 + 0.2) + 0.3
0.6000000000000001

>>> 0.1 + (0.2 + 0.3)
0.6

...浮点加法并不_完全_结合！
[来源]

一些提出的补救措施通常涉及对运算中涉及的所有操作数进行量化。但即使在这里，我们也必须小心确保诸如结合性之类的性质(a + b) + c = a + (b + c)能够促进可证明的决定论。如果您好奇的话，最近的研究论文中可能有更多关于这些补救措施的信息。
相关问题/评论：

SSF 模型在图像重建过程中遇到了问题。 第235章（评论）
 heng2020-anchor #229高频纹理的解码结果错误（评论）
关于评价 #54（评论）
使用 CUDA 预训练模型的奇怪行为 #47（评论）

Thank you very much for your reply, I will check the possible reasons.


Configuration Parameters: TorXakis.ini file

I would like a TorXakis.ini file to set my preferred parameter values.
Including deltaTime, Randomization settings etc...
In addition, I would like command line arguments, e.g., torxakis --paramX=ValX --paramY=ValY
Each parameter must have a specification whether or not this parameter can be changed at run-time.
The param command must divide the parameters into two parts: initialization-time and run-time changeable parameters.
When a parameter cannot be changed at run-time, an error is produced when the user tries to change it (using the param command).

related to #199 and #35

You you want a TorXakis.ini file (which we currently don't have), or are you ok with extending the .torxakis.yaml file? (which we have)

Extend the yaml file.....
Is old issue....

Remaining work:

[ ] command line arguments, e.g., torxakis --paramX=ValX --paramY=ValY
[ ] initialization-time and run-time changeable parameters
[ ] error handling for attempts to change init-time parameters with param command


BaseModel subclasses silently ignore properties named model_config

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
It seems that pydantic uses the model_config property to keep track of some internal states. However, when one tries to set a property called model_config instead of throwing up an error or warning, it silently skips it. Ideally, it should output a warning of some kind seeing how the name model_config is a natural variable name in domains such as machine learning.
Example Code
from pydantic import BaseModel

class Foo(BaseModel):
    foo: int

class Bar(BaseModel):
    model_config: Foo
    foo: Foo
    bar: int

foo = Foo(foo=12)
bar = Bar(model_config=foo, foo=foo, bar=13)

assert bar.foo == foo
assert bar.model_config == foo

Python, Pydantic & OS Version
pydantic version: 2.6.1
        pydantic-core version: 2.16.2
          pydantic-core build: profile=release pgo=true
                 install path: /home/lucas/.local/lib/python3.10/site-packages/pydantic
               python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
                     platform: Linux-6.5.0-17-generic-x86_64-with-glibc2.35
             related packages: typing_extensions-4.9.0 mypy-1.8.0
                       commit: unknown

@melembroucarlitos,
Thanks for reporting this.
Closing this as a duplicate of https://github.com/pydantic/pydantic/issues/8469, but I've added that issue to the 2.7.0 milestone, as this should be relatively easy to come up with a fix for, and it's certainly a request that many people have!