Catalog Resolution classpath to resolve schemas in same build

Hi,
I have a setup where I use a catalog resolver for schema resolution, with two projects:

project A contains a schema in classpath location 'xsd/common.xsd' and gets packaged including episode file and the catalog file
project B contains two schemas:

a.xsd importing just common.xsd
b.xsd importing common.xsd and a.xsd

The catalog file in project A maps http://www.example.com/xsd to classpath:xsd/ using REWRITE_SYSTEM. The imports in a.xsd and b.xsd of course use http://www.example.com/xsd/{common,a}.xsd as the schema locations.
The problems started when I added b.xsd: The classpath resolution for common.xsd works fine, but the import of a.xsd for b.xsd cannot be resolved.
With debugging output, I can see that he correctly tries to resolve a.xsd via maven:::!xsd/a.xsd while loading, but he can't find it.
So far I tried:

letting the CatalogResolutionClasspath extend configuration runtime
adding files('src/main/resources') to the CatalogResolutionClasspath

but none of these work, so I reimplemented the lookup procedure in MavenUriResolver directly in gradle as a task to take a look at the steps taken:
task tryResolve {
    doLast {
        def uri = new URI('maven:::!xsd/a.xsd')
        def schemeSpecific = uri.schemeSpecificPart
        println "URI: " + uri + " - schemeSpecific: " + schemeSpecific
        def dep = org.unbrokendome.gradle.plugins.xjc.resolver.DependencyResource.parse(schemeSpecific)
        println "dependency resource: " + dep
        def arts = configurations.jaxb.resolvedConfiguration.lenientConfiguration.getArtifacts(dep)
        println "Artifacts: " + arts
        def fAOne = arts.findAll { dep.classifier == null || it.classifier == dep.classifier }
        println "findAllOne: " + fAOne
        def fATwo = fAOne.findAll { dep.ext == null || it.ext == dep.ext }
        println "findAllTwo: " + fATwo
        if (dep.path) {
            println "Dependency has a path"
            def url = fATwo*.file*.toURI()*.toURL()
            println "URL" + url
        } else {
            println "Dependency does not have a path"
            println "URI: " + fATwo*.file*.toURI().first()
        }
    }
}

which leads to this output in both cases:
URI: maven:::!xsd/a.xsd - schemeSpecific: ::!xsd/a.xsd
dependency resource: org.unbrokendome.gradle.plugins.xjc.resolver.DependencyResource(xsd/a.xsd)
Artifacts: [common-xml.jar (project :common-xml)]
findAllOne: [common-xml.jar (project :common-xml)]
findAllTwo: [common-xml.jar (project :common-xml)]
Dependency has a path
URL: [file:/[...]/common-xml/build/libs/common-xml-0.1.jar]

When I substitute a.xsd in the task with the working common.xsd the output is the same, but of course that one can be found in the jar file by the URLClassLoader instantiated there.
Any ideas on how to get this working?
Thanks in advance.

Hi,
I would recommend to use a separate sub-path in the namespace for each project, so you can be more specific as to what should be resolved from the classpath and what not.
The only workaround I could imagine would be to use a different source set for the two schemas, so when the imports of b.xsd are resolved then a.xsd is already on the classpath.


extended unittests are failing

https://github.com/pytorch/vision/actions/runs/5941974400/job/16117254380
Failures start with 9c4f7389d0db7cfe7e8591ea920459673344aaa8, which is the first commit that used yesterdays (20230822) PyTorch nightly.
   =========================== short test summary info ============================
  FAILED test/test_extended_models.py::test_schema_meta_validation[vit_b_16] - AssertionError: assert not [(ViT_B_16_Weights.IMAGENET1K_V1, '_ops'), (ViT_B_16_Weights.IMAGENET1K_SWAG_E2E_V1, '_ops')]
  FAILED test/test_extended_models.py::test_schema_meta_validation[vit_b_32] - AssertionError: assert not [(ViT_B_32_Weights.IMAGENET1K_V1, '_ops')]
  FAILED test/test_extended_models.py::test_schema_meta_validation[vit_h_14] - AssertionError: assert not [(ViT_H_14_Weights.IMAGENET1K_SWAG_E2E_V1, '_ops'), (ViT_H_14_Weights.IMAGENET1K_SWAG_LINEAR_V1, '_ops')]
  FAILED test/test_extended_models.py::test_schema_meta_validation[vit_l_16] - AssertionError: assert not [(ViT_L_16_Weights.IMAGENET1K_V1, '_ops'), (ViT_L_16_Weights.IMAGENET1K_SWAG_E2E_V1, '_ops')]
  FAILED test/test_extended_models.py::test_schema_meta_validation[vit_l_32] - AssertionError: assert not [(ViT_L_32_Weights.IMAGENET1K_V1, '_ops')]
  ====== 5 failed, 658 passed, 1 skipped, 430 warnings in 511.39s (0:08:31) ======

It is not obvious from the error message, but the failure here comes from the fact the number that we have on record, e.g.
https://github.com/pytorch/vision/blob/11e49de410ec84ec669293a91dfaa13a53c9bc47/torchvision/models/vision_transformer.py#L392
no longer matches what we calculate.
cc @seemethere

Looks the same as https://github.com/pytorch/vision/issues/7349. Skip the test :(

If it is flakiness, the error is huge:
from common_extended_utils import get_ops
from torchvision import models

for model_fn, weights in [
    (models.vit_b_16, models.ViT_B_16_Weights.IMAGENET1K_V1),
    (models.vit_b_16, models.ViT_B_16_Weights.IMAGENET1K_SWAG_E2E_V1),
    (models.vit_b_32, models.ViT_B_32_Weights.IMAGENET1K_V1),
    (models.vit_h_14, models.ViT_H_14_Weights.IMAGENET1K_SWAG_E2E_V1),
    (models.vit_h_14, models.ViT_H_14_Weights.IMAGENET1K_SWAG_LINEAR_V1),
    (models.vit_l_16, models.ViT_L_16_Weights.IMAGENET1K_V1),
    (models.vit_l_16, models.ViT_L_16_Weights.IMAGENET1K_SWAG_E2E_V1),
    (models.vit_l_32, models.ViT_L_32_Weights.IMAGENET1K_V1),
]:
    expected = weights.meta["_ops"]
    actual = get_ops(model=model_fn(weights=weights), weight=weights)
    exact_float_match = expected == actual

    print(f"{model_fn.__name__} / {weights.name}: {actual=}, {expected=}, {exact_float_match=}")

vit_b_16 / IMAGENET1K_V1: actual=16.849, expected=17.564, exact_float_match=False
vit_b_16 / IMAGENET1K_SWAG_E2E_V1: actual=49.348, expected=55.484, exact_float_match=False
vit_b_32 / IMAGENET1K_V1: actual=4.363, expected=4.409, exact_float_match=False
vit_h_14 / IMAGENET1K_SWAG_E2E_V1: actual=862.961, expected=1016.717, exact_float_match=False
vit_h_14 / IMAGENET1K_SWAG_LINEAR_V1: actual=161.884, expected=167.295, exact_float_match=False
vit_l_16 / IMAGENET1K_V1: actual=59.647, expected=61.555, exact_float_match=False
vit_l_16 / IMAGENET1K_SWAG_E2E_V1: actual=310.346, expected=361.986, exact_float_match=False
vit_l_32 / IMAGENET1K_V1: actual=15.255, expected=15.378, exact_float_match=False

Maybe something was merged into core that actually reduced the number significantly?

It is definitely not flakiness this time. For some reason, we no longer have values for aten.bmm, which is what is causing the drop. The other ops are unaffected.

We are currently tracking the following ops
https://github.com/pytorch/vision/blob/c486bb14ad386257b4125ae08a11c3f87b5f41fc/test/common_extended_utils.py#L143-L153
We need to add an
aten._scaled_dot_product_flash_attention: scaled_dot_product_flash_attention_flop

entry there. The function scaled_dot_product_flash_attention_flop needs to count the flops of the underlying kernel.
That is quite some work. @NicolasHug are you ok with me disabling this subtest for the offending models for now and fix this after release?

yeah

@pmeier,  sorry for hijacking the thread as I could not contact you anyway. Could you please look into this ?


Fix TypeError when Bootstrap is included in head

I've added waiting for domContentLoaded event before extending jQuery. That's one possible solution to be sure, that data attributes of body can be read.
Fixes #31202
Preview: https://deploy-preview-32024--twbs-bootstrap.netlify.app/

It would be useful having some tests mocking getjQuery or jquery in window object, so that the uncovered parts are covered.
I'm struggling a bit with mocking these parts.

We still have the problem in event-handler.js, when calling getjQuery().

I would prefer a way like this: https://github.com/twbs/bootstrap/commit/ff1fa49b88e1c48de603f1613b04650d2f8354d0
It's much cleaner and there shouldn't be no problems. If we'd implement it with event listener, the problem could be, that the user has to wait for DOMContentLoaded at some places too.

it seems something decreased the minimum coverage required

Do you know, if coverage also includes js-test-jquery?

The coverage doesn't include the jQuery part.
I don't think it's really necessary since jQuery is a very small part of our codebase and the main goal of v5 is to work without jQuery

Okay, still not familiar enough with bootstrap codebase. I simply overlooked the /* istanbul ignore if */ statements.

@tidusIO apart from my comment above, can you please make a CodePen with the scripts in head so that we can test?

Yes, I've replied your comment. I can, but probably not before Wednesday.

NVM I tested it myself and it works as expected, thanks for the PR!

Bad decision. Jquery plugins should be loaded in developer defined order: there is convention that this order is maintained by script loading.   How to control this order after you broke it with domContentLoaded  ? What if every jquery plugin start doing it this way?

Rly like last comment. We works with sharepoint onprem, and have a problem there. SP have own event model, many usefull events trigered before DOMContentLoaded without access to jquery bootstrap plagins. We spand a time for only understand that plagins extend performed in postponed format.


Fix unmount

fix for issue with unmount not running useEffect cleanup: https://github.com/wojtekmaj/enzyme-adapter-react-17/issues/12

Would be helpful if this pull request was merged :)

we are using this library as well and it would be very useful

@gisaklement I started discussing this change in a PR that adds official React 17 adapter this package is based on. Let's see how it goes.

@gisaklement I started discussing this change in a PR that adds official React 17 adapter this package is based on. Let's see how it goes.

Thank you so much!! I appreciate it 🙂

Looks like we won't get an official response anytime soon. Screw this then. @Iskander508 Please kindly rebase your branch off current master, or allow edits from maintainers, so I could check if this doesn't break anything, since tests were added recently.

Looks like we won't get an official response anytime soon. Screw this then. @Iskander508 Please kindly rebase your branch off current master, or allow edits from maintainers, so I could check if this doesn't break anything, since tests were added recently.

Edits should be allowed already

rebased

Thanks so much.
Interestingly this doesn't seem to affect neither positively nor negatively any unit tests originally added to Enzyme.
Let's go for it.

Again, thank you very much 🙂


Create a 'Breakpoints' Gt Class View.

It'd be useful to see at a glance if a class has some breakpoints and on what methods.
Seems like this can be a gtClassView with the result of some search filters.
Icing on the cake would be to have some functionality through UI to comment or remove said breakpoints, either individually or in a batch refactoring.

I have custom views defined in this repo. Unclear if most people would find this interesting or not. Open to opening PR for this, if not going to close as stale


adomani: test a trivial addition

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

maintainer merge

The corresponding mathlib PR implementing maintainer merges split by topics has been merged, fixed and tested!


Dedicated servers broken...

Basically everytime I try to join my own server or someone trys to join my server it works fine until someone else joins. There's no error message and usually the person who joins second gets a infinite loading screen. The match works just fine for the solo person until another person joins.
Here's the startup args I'm running : '+setplaylist ffa +mp_gamemode ffa +map mp_wargames +max_players 12 +pilot_health_multiplier 1.25 +boosts_enabled 1 +earn_meter_pilot_overdrive 0 +earn_meter_pilot_multiplier 0'
I haven't really changed anything else that would cause this to my knowledge besides the autoexec_ns_server.cfg which I changed the 'ns_should_return_to_lobby' to '0' which should make it where after the match ends it just goes straight to the next match.
Any ideas on what could fix this or what could be happening? My ultimate goal is to make a few servers that are on gamemodes like FFA and CTF that rotate through different maps after each match ends.

to be fair about to start another issue issue request because I can't connect to my own server....

to be fair about to start another issue issue request because I can't connect to my own server....

If you have multiple maps put in the startup args, then it won't let you connect. Or if you are connecting from the same PC then that could be causing the issue

yeah it is the same PC is that the issue?

I mean I see people running titanfall 2 on linux maybe I can get a dedicated server running on my server

Hello, I'm running 1.4.0 (latest release at the time) and I have the same issue
First person connecting it works just fine, then anyone who connects after that gets infinite loading. The server also crashes on its own after about three or four minutes, without doing anything in particular.
Why was the issue closed - is there a fix or was it fixed? I do not see a commit referenced
Have a nice day
+setplaylist private_match +ns_server_name "Test server" +mp_gamemode ffa +map mp_eden

For anyone who stumbles upon this thread, the temporary fix i found is to never indicate a map in the launch arguments and to supply it later, by writing map mp_eden in the console for instance.

+map is currently unstable. Do not use it.


How to decode traceid read from kafka

I'm using kafka in Jaeger System, and I got traceid like "AAAAAAAAAABx3MgiIQRHEg==",  span_id like "jkVS3tZkQKM="  from Kafka
There are some Issues say that the traceId can decode by base64,  but I can't
So  how to decode traceId With Python or C++

What are "some issues"? This is standard base64.

What are "some issues"? This is standard base64.

emmmmm....
issues: Why is the data received from kafka like this? question, and how to decode trace_id from kafka protobuf data? question
But I get maybe wrong result decoding with  Base64 using https://www.base64decode.org/
Input: AAAAAAAAAABx3MgiIQRHEg==
output: qÜÈ"!G

Why do you think it's wrong? If you're trying to get to a hex ID, you need to decode to a byte array first and the convert bytes to hex. The online converter tries to interpret those bytes as string.

Why do you think it's wrong? If you're trying to get to a hex ID, you need to decode to a byte array first and then convert bytes to hex. The online converter tries to interpret those bytes as string.

I still don't know how the traceId in kafka was produced.....
You said ," you need to decode to a byte array first and then convert bytes to hex"
Is it means that traceId gotton from Jaeger UI is hex ?
For example, a traceid , "abe33738f1825469",  will first decode with hex, then encode with base64,
so it can became " AAAAAAAAAABx3MgiIQRHEg=="  ?

abe33738f1825469

you first convert it into byte array, [0xAB, 0xE3, 0x37, ...], and then encode this byte array with base64.

Btw, are you using JSON encoding with Kafka?  We use binary protobuf, which is much more efficient.

Btw, are you using JSON encoding with Kafka? We use binary protobuf, which is much more efficient.

“JSON encoding”, you mean the parameter  --kafka.producer.encoding=json of jaeger-collcoter ?
I do use JSON encoding
Maybe when I knew the protobuf and Jaeger I will try using protobuf
but now I'm noob....
Thank you very much
Let me try the process that you said of decode...

abe33738f1825469

you first convert it into byte array, [0xAB, 0xE3, 0x37, ...], and then encode this byte array with base64.

emmmmm....
I'm trying to use java to simulate the process you said . But maybe I do something wrong... So could you please help me find what's the problem?
import java.io.* ;
import java.util.Arrays;
import java.util.Base64;

public class HelloWorld {
    public static void main(String []args) {
        byte[] ascii = "83e968fc89ec0038".getBytes();
        String byteArray = Arrays.toString(ascii);
	System.out.println("contents of byte array in default encoding: " + byteArray);
	String base64Str = Base64.getEncoder().encodeToString(ascii);
	System.out.println("base 64 code: " + base64Str);
    }
}

>>>
contents of byte array in default encoding: [56, 51, 101, 57, 54, 56, 102, 99, 56, 57, 101, 99, 48, 48, 51, 56]
base 64: ODNlOTY4ZmM4OWVjMDAzOA==

the traceid I got from Jaeger UI  is "83e968fc89ec0038", and from the java code I got base64 code  "ODNlOTY4ZmM4OWVjMDAzOA==", which can't be found in Kafka msg
the traceId in kafka msg more like "AAAAAAA........."

byte[] ascii = "83e968fc89ec0038".getBytes();
This returns a byte for every character in the string. You need to convert the string to bytes using hex encoding.

byte[] ascii = "83e968fc89ec0038".getBytes();
This returns a byte for every character in the string. You need to convert the string to bytes using hex encoding.

There's gotta be someway to complaint someone like you who perverted my mind!
Look at this !  ids.go
// TraceIDFromString creates a TraceID from a hexadecimal string
func TraceIDFromString(s string) (TraceID, error) {
	var hi, lo uint64
	var err error
	switch {
	case len(s) > 32:
		return TraceID{}, fmt.Errorf("TraceID cannot be longer than 32 hex characters: %s", s)
	case len(s) > 16:
		hiLen := len(s) - 16
		if hi, err = strconv.ParseUint(s[0:hiLen], 16, 64); err != nil {
			return TraceID{}, err
		}
		if lo, err = strconv.ParseUint(s[hiLen:], 16, 64); err != nil {
			return TraceID{}, err
		}
	default:
		if lo, err = strconv.ParseUint(s, 16, 64); err != nil {
			return TraceID{}, err
		}
	}
	return TraceID{High: hi, Low: lo}, nil
}

That 's what you say standard base64
Fuck!

Your original question was how to convert an ID from JSON:

I'm using kafka in Jaeger System, and I got traceid like "AAAAAAAAAABx3MgiIQRHEg=="

This ^^ is base64 encoding.  TraceIDFromString() converts from hex encoding.

Your original question was how to convert an ID from JSON:

I'm using kafka in Jaeger System, and I got traceid like "AAAAAAAAAABx3MgiIQRHEg=="

This ^^ is base64 encoding. TraceIDFromString() converts from hex encoding.

So what is the difference ？
I want to covert an id  from Kafka JSON to  an hex ID which  in jaeger UI
And then I  ask how an id  convert from original ID to hex ID
You said that "If you're trying to get to a hex ID, you need to decode to a byte array first and then convert bytes to hex. The online converter tries to interpret those bytes as string"
But from the code in ids.go, I  know that It's not stand hex convert.
To trans Hex ID to base64 ID, or trans Base64 ID to Hex ID, I must transfer some fucntions in ids.go from Go to Python or Java or any language I need
That‘’s not ** standard** at all

any news here? references to converters would be great..

It May be too late, but hopefully this will help
maven dependency
<dependency>
    <groupId>commons-codec</groupId>
    <artifactId>commons-codec</artifactId>
    <version>1.15</version>
</dependency>

Imports
import org.apache.commons.codec.binary.Base64;
import static org.apache.commons.codec.binary.Hex.encodeHex;

code
final String encoded = "PK0fIykrpiGLttuOy9Y2FA==";
final String traceid = new String(encodeHex(Base64.decodeBase64(encoder)));

I guess the traceid/spanid is in string format, if it is bytestring then use .toByteArray()


[REEF-418]: Group Communication breaks down on actual cluster

After the recent merge (see REEF-260), we found that once we run PipelineBroadcastAndBroadcast Reduce on actual cluster it fails while if we run it locally it passes. We found the issue to be in the WritableNetworkService where while registering observer we map it to a particular local IP address rather than any.
The aim of this JIRA is to make this one line change in the WritableNetworkService.
This addressed the issue by

Registering observer in DefaultRemoteManager with IpAddress.Any in WritableNetworkService
JIRA: REEF-418

I'll file the JIRA and update the code with a //TODO while merging it.


container overflow

looks like something needs to be clearfix'd perhaps?

this occurs on mobile devices only.  tried using default <html> as well

the child of the card-container doesn't honor/compute it's height correctly. e.g., the card-container may be 135px computed height, yet the child jp-container computes a 200px. height

@cdaringe This issue is probably related to original card, but not angular-card directives

Good point.  Thx


Healthcheck UI on the /config-wizard flow

Implement the healthcheck UI as discussed here

Should this be on Fides? or Fidesplus?

I can't find this indicator when using most recent build. Can somebody else test and verify this is working? @seanpreston

@rsilvery are you still having problems seeing this?

Nope!


[SPARK-21786][SQL] The 'spark.sql.parquet.compression.codec' and 'spark.sql.orc.compression.codec' configuration doesn't take effect on hive table writing

[SPARK-21786][SQL] The 'spark.sql.parquet.compression.codec' and 'spark.sql.orc.compression.codec' configuration doesn't take effect on hive table writing
What changes were proposed in this pull request?
Pass ‘spark.sql.parquet.compression.codec’ value to ‘parquet.compression’.
Pass ‘spark.sql.orc.compression.codec’ value to ‘orc.compress’.
How was this patch tested?
Add test.

Can one of the admins verify this patch?

Could you add tests? Probably, you could insert some data then check if the data compressed by listing up files in temp dir?

cc @maropu
I have added the test. However, all of my local use cases do not work properly, so I'm not sure if the new use case will pass, but I will always be concerned.

@gatorsmile Is it worth fixing this? If so, could you trigger tests?

ok to test

Test build #81813 has started for PR 19218 at commit 4e70fff.

Test build #81813 has finished for PR 19218 at commit 4e70fff.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81813/
Test FAILed.

@maropu @fjh100456 If the issue is true, we should fix it for sure. However, the PR description must be wrong. If this issue exists, it should be applicable to both partitioned and non-partitioned ORC/Parquet tables. cc @dongjoon-hyun I think you might be interested in ORC side.

@gatorsmile

Non-partitioned tables do not have this problem, 'spark.sql.parquet.compression.codec' can take effect normally, because the process of writing data differs from that of a partitioned table.
ORC does not have a configuration similar to 'spark. sql.* ', but can only use ' orc. compress ' which may not be a spark configuration.

Test build #81815 has started for PR 19218 at commit 3f022f9.

Test build #81815 has finished for PR 19218 at commit 3f022f9.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81815/
Test FAILed.

Test build #81816 has started for PR 19218 at commit 6d77bf9.

Test build #81816 has finished for PR 19218 at commit 6d77bf9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81816/
Test FAILed.

Test build #81820 has started for PR 19218 at commit 42aca3d.

Test build #81820 has finished for PR 19218 at commit 42aca3d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81820/
Test PASSed.

Merged build finished. Test PASSed.

Thank you for pinging me, @gatorsmile.

@dongjoon-hyun Thank you very much,  I'll fix it now.

Test build #81839 has started for PR 19218 at commit 5cbe999.

Test build #81839 has finished for PR 19218 at commit 5cbe999.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81839/
Test FAILed.

Test build #81840 has started for PR 19218 at commit 732266c.

Test build #81840 has finished for PR 19218 at commit 732266c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81840/
Test FAILed.

Test build #81841 has started for PR 19218 at commit c7ff62c.

@dongjoon-hyun
A problem has been encountered, There are two ways to specify the compression format:

CREATE TABLE Test(id int) STORED AS ORC TBLPROPERTIES ('orc.compress'='SNAPPY');
set orc.compress=ZLIB;
If the table already has been specified a compressed format when it was created, and then specified another compression format by setting 'orc.compress', the latter will take effect.

So whether the spark side should not have the default value, we can distinguish by 'undefined'; or discard this change, and explain in the document that 'spark.sql.parquet.compression.codec' for partitioned tables does not take effect, and 'spark.sql.orc.compression.codec ' is not valid for hive tables. Or your other better solution.

Test build #81841 has finished for PR 19218 at commit c7ff62c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81841/
Test PASSed.

I see. If you set spark.sql.hive.convertMetastoreParquet to false, you will also hit the issue for non-partitioned table.

@fjh100456 We have priority for three different inputs. Here, you just consider one of three. Please also add the extra checks. Hopefully, @dongjoon-hyun can help you answer your questions. He just finished the work in https://github.com/apache/spark/pull/19055

Sorry, guys. I've been away from keyboard since last Friday night. I'll be back on next Tuesday (PST).

Encounter two problems:

I tried to fix it in the order of 'compression' > 'parquet.compression' > 'spark.sql.parquet.compression. codec', but found 'parquet.compression' may come from a user 'set' command or from a table property which is specified on creation, so which one should take precedence?

If I change the priorities, it seems partitioned tables and non-partitioned tables are all need to be fixed, maybe I have to moves the code forward, for example, to 'Insertintohadoopfsrelationcommand.scala'.

I would appreciate it if you could give me some advice. @gatorsmile  @dongjoon-hyun

You can get it from the table metadata table: CatalogTable

Insertintohadoopfsrelationcommand.scala  is for data source tables. We only have the issues for Hive table writing, right?

Thanks for your review. @gatorsmile
In the first question I mean that ‘parquet.compression’ can be found in the table: Tabledesc (maybe similar with catalogtable), and can also be found in sparkSession.sessionState.conf(set by user through the command set parquet.compression=xxx), which one should take precedence?
This issue was originally only related to hive table writing,  but after fix the priority, it was found that non-partitioned tables did not take the right precedence, and non-partitioned tables writing will not enter InsertIntoHiveTable. Insertintohadoopfsrelationcommand.scala is really not a proper place, is there any place that can solve both of partitioned tables and non-partitioned tables?

Table-level metadata has the highest priority. In the table-level metadata, compression has a higher priority than parquet.compression. If the is not specified, we need to check the session-level config spark.sql.parquet.compression.codec.

You just need to fix InsertIntoHiveTable . It is not related to Insertintohadoopfsrelationcommand.scala. In your test cases, you just need to set spark.sql.hive.convertMetastoreParquet to false

Test build #81989 has started for PR 19218 at commit 384ee04.

Test build #81989 has finished for PR 19218 at commit 384ee04.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81989/
Test FAILed.

Test build #81992 has started for PR 19218 at commit 8c92074.

Test build #81992 has finished for PR 19218 at commit 8c92074.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81992/
Test FAILed.

Test build #81993 has started for PR 19218 at commit dd5060a.

Test build #81993 has finished for PR 19218 at commit dd5060a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81993/
Test FAILed.

Test build #81994 has started for PR 19218 at commit d427df5.

Test build #81994 has finished for PR 19218 at commit d427df5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81994/
Test FAILed.

Test build #81995 has started for PR 19218 at commit 35cfa01.

Test build #81995 has finished for PR 19218 at commit 35cfa01.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81995/
Test FAILed.

Test build #82001 has started for PR 19218 at commit 5387497.

Test build #82001 has finished for PR 19218 at commit 5387497.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82001/
Test PASSed.

Hi, @fjh100456 . I left a few comments.

@dongjoon-hyun
Thank you very much, I'll fix them as soon as possible.

Test build #82240 has started for PR 19218 at commit 676d6a7.

Test build #82240 has finished for PR 19218 at commit 676d6a7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
    case class TableCompressionConf(name: String, codeC: String)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82240/
Test FAILed.

Test build #82244 has started for PR 19218 at commit ae1da8f.

Test build #82244 has finished for PR 19218 at commit ae1da8f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82244/
Test FAILed.

Test build #82260 has started for PR 19218 at commit fd73145.

Test build #82260 has finished for PR 19218 at commit fd73145.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82260/
Test FAILed.

Test build #82261 has started for PR 19218 at commit 7615939.

Test build #82261 has finished for PR 19218 at commit 7615939.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82261/
Test PASSed.

@dongjoon-hyun  @gatorsmile
I'd fix them. Could you help me to review it again? Thanks.

LGTM

Test build #82579 has started for PR 19218 at commit 90cbcb3.

Test build #82580 has started for PR 19218 at commit dd6d635.

Test build #82579 has finished for PR 19218 at commit 90cbcb3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82579/
Test FAILed.

Test build #82580 has finished for PR 19218 at commit dd6d635.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82580/
Test PASSed.

cc @gatorsmile @dongjoon-hyun
Is it ok now?

Test build #82660 has started for PR 19218 at commit 4fe8170.

Test build #82660 has finished for PR 19218 at commit 4fe8170.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82660/
Test FAILed.

Test build #82772 has started for PR 19218 at commit aa31261.

Test build #82772 has finished for PR 19218 at commit aa31261.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82772/
Test FAILed.

Test build #82784 has started for PR 19218 at commit dfb36d9.

Test build #82784 has finished for PR 19218 at commit dfb36d9.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82784/
Test FAILed.

Test build #82785 has started for PR 19218 at commit c4801f6.

Test build #82785 has finished for PR 19218 at commit c4801f6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82785/
Test FAILed.

Test build #82800 has started for PR 19218 at commit 105e129.

Test build #82800 has finished for PR 19218 at commit 105e129.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82800/
Test PASSed.

cc @gatorsmile @dongjoon-hyun
Is it ok now?

Can one of the admins verify this patch?

@gatorsmile
I'd tested the performance of 'uncompressed', 'snappy', 'gzip' compression algorithm for parquet, the input data volume is 22MB, 220MB, 1100MB, respectively run 10 times, finally 'snappy' in several cases are more excellent.
The test results are as follows:(TimeUnit: ms)

@fjh100456 What is the workload you ran?

@gatorsmile
Thank you very much.I'll fix all the problem later.For more，I'm not very clear what you mean by 'workload', I changed the test to a benchmark in this two days, and the code and the results are as follows:
class InsertHiveTableBenchmark extends SQLTestUtils {
  lazy val spark = SparkSession.builder
    .enableHiveSupport()
    .master("local[1]")
    .appName("microbenchmark")
    .config("spark.sql.warehouse.dir", "file:/E:/study/temp/warehouse")
    .config("hive.exec.dynamic.partition.mode", "nonstrict")
    .getOrCreate()

  /** Runs function `f` with different format. */
  private def runBenchmark(
    name: String, cardinality: Long, tables: List[String])(f: String => Unit): Unit = {
    val benchmark = new Benchmark(name, cardinality)
    val numIters = 5

    tables.map { table =>
      benchmark.addCase(s"To $table", numIters)(iter => f(table))
    }

    benchmark.run()
  }

  private val dataSourceName: String = "projdata"

  private def getTableByFormatCompression(format: String, compressionCodec: String): String = {
    s"${dataSourceName}_${format}_${compressionCodec}"
  }

  private def createTable(format: String, compressionCodec: String): String = {
    val tableName = getTableByFormatCompression(format, compressionCodec)
    val compressionName = getCompressionNameByFormat(format)
    spark.sql(
      s"""CREATE TABLE if not exists $tableName(
      /*---About 74 fields, Forgive me for not being able to show all the fields
             because being banned by the company---/*)
         |PARTITIONED BY (p_provincecode int)
         |STORED AS $format
         |TBLPROPERTIES('$compressionName'='$compressionCodec')""".stripMargin)
    tableName
  }

  private def insertOverwriteTable(tableName: String, dataSource: String): Unit = {
    spark.sql(s"insert overwrite table $tableName PARTITION(p_provincecode=510000) select * from $dataSource")
  }

  private def withCachedTable(sizeInMb: Int)(f: (String, Long) => Unit): Unit = {
    val size = 22 //textfile data size in MB
    val times = if (sizeInMb <= size) 1 else sizeInMb / size
    val unions = (0 until times).map(_ => s"select * from $dataSourceName where p_provincecode=510000")
    val cache = spark.sql(unions.mkString(" union all ")).cache()
    val tempViewName = s"${dataSourceName}_cache"
    cache.createTempView(tempViewName)
    try {
      val numRecords = cache.count()
      f(tempViewName, numRecords)
    } finally {
      spark.sqlContext.dropTempTable(tempViewName)
      cache.unpersist()
    }
  }

  private def getCompressionNameByFormat(format: String): String = {
    format.toLowerCase match {
      case "parquet" => "parquet.compression"
      case "orc" => "orc.compress"
      case _ => throw new Exception(s"Invalid format $format")
    }
  }

  private def getCompressionCodesByFormat(format: String): List[String] = {
    format.toLowerCase match {
      case "parquet" => List("UNCOMPRESSED", "SNAPPY", "GZIP")
      case "orc" => List("NONE", "SNAPPY", "ZLIB")
      case _ => throw new Exception(s"Invalid format $format")
    }
  }

  private def performance(dataSizeInMB: Int, format: String): Unit = {
    withCachedTable(dataSizeInMB) { case (cacheName, numRecords) =>
      val tables = getCompressionCodesByFormat(format).map(codec => createTable(format, codec))
      withTable(tables:_*) {
        runBenchmark(s"Inert overwrite ${dataSizeInMB}MB to hive table", numRecords, tables){ tableName =>
          insertOverwriteTable(tableName, cacheName)
        }
      }
    }
  }

  test("Insert 22MB textfile datasource into parquet hive table") {
    performance(22, "parquet")
    /*
    Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13 on Windows 7 6.1
    Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
    Inert overwrite 22MB to hive table:      Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
    ------------------------------------------------------------------------------------------------
    To projdata_parquet_UNCOMPRESSED      2043 / 2509          0.0       45247.5       1.0X
    To projdata_parquet_SNAPPY         1684 / 2806          0.0       37291.0       1.2X
    To projdata_parquet_GZIP           2066 / 2700          0.0       45759.0       1.0X
    */
  }

  test("Insert 220MB textfile datasource into parquet hive table") {
    performance(220, "parquet")
    /*
    Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13 on Windows 7 6.1
    Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
    Inert overwrite 220MB to hive table:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
    ------------------------------------------------------------------------------------------------
    To projdata_parquet_UNCOMPRESSED     9156 / 11150          0.0       20280.9       1.0X
    To projdata_parquet_SNAPPY        8949 / 17424          0.1       19823.1       1.0X
    To projdata_parquet_GZIP         14563 / 19023          0.0       32259.1       0.6X
    */
  }

  test("Insert 1100MB textfile datasource into parquet hive table") {
    performance(1100, "parquet")
    /*
    Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13 on Windows 7 6.1
    Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
    Inert overwrite 1100MB to hive table:    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
    ------------------------------------------------------------------------------------------------
    To projdata_parquet_UNCOMPRESSED    44395 / 53093          0.1       19667.8       1.0X
    To projdata_parquet_SNAPPY       40974 / 42340          0.1       18152.2       1.1X
    To projdata_parquet_GZIP         57086 / 58069          0.0       25290.0       0.8X
    */
  }

  test("Insert 22MB textfile datasource into orc hive table") {
    performance(22, "orc")
    /*
    Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13 on Windows 7 6.1
    Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
    Inert overwrite 22MB to hive table:      Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
    ------------------------------------------------------------------------------------------------
    To projdata_orc_NONE               1440 / 1495          0.0       31887.3       1.0X
    To projdata_orc_SNAPPY             1467 / 1550          0.0       32488.7       1.0X
    To projdata_orc_ZLIB               1546 / 1649          0.0       34253.6       0.9X
    */
  }

  test("Insert 220MB textfile datasource into orc hive table") {
    performance(220, "orc")
    /*
    Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13 on Windows 7 6.1
    Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
    Inert overwrite 220MB to hive table:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
    ------------------------------------------------------------------------------------------------
    To projdata_orc_NONE              9108 / 10050          0.0       20174.3       1.0X
    To projdata_orc_SNAPPY             8937 / 9052          0.1       19795.1       1.0X
    To projdata_orc_ZLIB              9906 / 10236          0.0       21943.0       0.9X
    */
  }

  test("Insert 1100MB textfile datasource into orc hive table") {
    performance(1100, "orc")
    /*
    Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13 on Windows 7 6.1
    Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
    Inert overwrite 1100MB to hive table:    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
    ------------------------------------------------------------------------------------------------
    To lte_cm_projdata_txt_orc_NONE             47655 / 48881          0.0       21112.0       1.0X
    To lte_cm_projdata_txt_orc_SNAPPY           44012 / 45273          0.1       19498.2       1.1X
    To lte_cm_projdata_txt_orc_ZLIB             49044 / 50509          0.0       21727.2       1.0X
    */
  }
}

Can one of the admins verify this patch?

@fjh100456 Any progress?

@fjh100456 My concern is the size change could causes the statistics changes. However, this is a bug fix. We should fix it. Thanks!

In the case xiao suggested above, users usally re-run ANALYZE TABLE?

I will change the code with the suggestion of @gatorsmile ,it's a little busy this days.I will do it tomorrow.

@gatorsmile @maropu  Does it look better now? About statistic issue, is there any suggestion?
@SparkQA Please start test, thanks.

ok to test

Test build #85154 has started for PR 19218 at commit d779ee6.

Test build #85154 has finished for PR 19218 at commit d779ee6.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ParquetOptions(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85154/
Test FAILed.

Test build #85173 has started for PR 19218 at commit 0cb7b7a.

Test build #85173 has finished for PR 19218 at commit 0cb7b7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85173/
Test PASSed.

@gatorsmile Could you help to review it? Thanks very much!

Could you also add another test scenario? For the existing Hive tables (created by Hive), does our Spark respect it? Do we use the existing compression configuration?

@gatorsmile
I'd test manully. When table-level compression not configured, it always take the session level compression, and ignore the existing file compression. Seems like a bug, however, table files with multiple compressions do not affect the reading and writing.
Is it ok to add a test to check reading and writing when there are multiple conpressions in the existing table files?

What are multiple expressions?

I mean sevral files with different compressioncodcs under the same table directory, like below:

Test build #85334 has started for PR 19218 at commit 78e0403.

Test build #85335 has started for PR 19218 at commit 7804f60.

Test build #85334 has finished for PR 19218 at commit 78e0403.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85334/
Test PASSed.

Merged build finished. Test PASSed.

Test build #85335 has finished for PR 19218 at commit 7804f60.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ParquetOptions(

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85335/
Test PASSed.

Merged build finished. Test PASSed.

Test build #85340 has started for PR 19218 at commit 52cdd75.

Test build #85340 has finished for PR 19218 at commit 52cdd75.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85340/
Test FAILed.

Merged build finished. Test FAILed.

@fjh100456 Yeah. Please also add the test cases with the table containing mixed compression codec. Thanks!
I have some comments about your fix. See my commit:
https://github.com/apache/spark/commit/d8fbdaead5262fa1a04986831ca7a5b01158c7f7
I will review your fix later

I'd finished to write the test case with the table containing mixed compression codec. But maybe I'd made a mistake, the original branch was deleted mistakenly, I will closed this PR and create another PR. Sorry.

Please go to #20087


fix: removing warning

As the title.
I use the github dev website to submit this, soo.... pre-commit might complain.
Close #229

. pre-commit might complain.

Good thing we have pre-commit.ci enabled 😄

Thank you a lot @RobPasMue !! :)


🚀 Feature Request: Deployment Rollback

Describe the solution
Hello and first of all a huge thank you for building this groundbreaking platform!
I want to file a quick suggestion to enhance Wrangler CLI. So more stuff could be done without the need to access the Dashboard (especially given the fact that not everyone in the company has an access to the Dashboard, which is increasingly the case in bigger companies).
Deployment Rollback
This feature is already available in the Dashboard, so porting it to Wrangler shouldn't be a big effort, but it can make a big impact to help teams with organizing their workflows to easily recover from some unwanted deployment regressions.
The API could be as simple as:
wrangler pages deployment rollback --project-name myproject --to-id 1xx11xx1x1xx1x1xx1x1
where --to-id flag specifies the id of the deployment to roll back to.
Thanks!

Thanks for the feature request! I'll keep this open for now to track this. Cheers!

@awthwathje from my experience it doesn't. i'm getting the following error when I run wrangler rollback in the root of my pages project and pass it correct deployment id (took it from cf dashboard)
  This Worker does not exist on your account. [code: 10007]

@vorant94 — for Pages, you need to rollback using the Cloudflare dashboard: https://developers.cloudflare.com/pages/configuration/rollbacks/
The wrangler rollback works with Workers.
Workers now support static assets, so you may be able to use Workers for your project, if you need to use this: https://developers.cloudflare.com/workers/static-assets/compatibility-matrix/
More generally, working towards bringing Workers and Pages closer together in functionality, know that it is confusing to have both and to get this error.
Seems like when you run wrangler rollback in the directory of a Pages project, we could throw a more helpful error here.

@irvinebroque yeah, after I got the error from above I realized that dashboard is the only option for now
It's not fitting my case unfortunately, because my initial case was to build a GH workflow that goes something like this "build->publish->e2e test->rollback if published, but e2e failed". And it is possible only with rollbacks via cli (telling all of this just for the sake of sharing the case, where this feature might be needed)
Currently I'm thinking about alternatives

@vorant94 try deploying your project to Workers, curious what you run into.
And talk to @aninibread about that workflow — would love to be able to handle more of that ourselves for you as part of builds.


1.3.0 does not seem to be updated accordingly

When I install the package as latest version (1.3.0) from npm, I see the ReadMe file is updated; but I see the only two index.js files in the directories node_modules/remark-custom-container/dist and node_modules/remark-custom-container/dist/esm which seem not to updated according to latest changes.

Thanks for reporting, it doesn't include latest dist.
It'll fixed in https://github.com/koka831/remark-custom-container/pull/155 and I'll release it in 1.3.1.

@talatkuyuk Could you try https://www.npmjs.com/package/remark-custom-container/v/1.3.1 ?

Done. It seems to be updated now, thanks for quick action 👍


Nidaqmx

New Nidaqmx Node based on the official python wrapepr from ni.

Looks like there is some code to support analog output channels.. is this just not finished yet?

For the first round it is finished. Until the enxt bug.
The scenario is: coninuous acquisition on input. And episodic play some signal on output (at the same sample rate). So that feed my needs for the moments.
We should also add digital input/output but this will be for next time.

Error running the example:
Traceback (most recent call last):
  File "c:\users\lukec\pyacq\pyacq\devices\ni_daqmx.py", line 232, in run
    nb_sample = aitask.in_stream.readinto(raw_data_flat)
  File "C:\Anaconda3\lib\site-packages\nidaqmx\_task_modules\in_stream.py", line 1992, in readinto
    self.timeout)
  File "C:\Anaconda3\lib\site-packages\nidaqmx\_task_modules\read_functions.py", line 583, in _read_raw
    ctypes.byref(number_of_bytes_per_sample), None)
ctypes.ArgumentError: argument 2: <class 'TypeError'>: wrong type

You need this: https://github.com/ni/nidaqmx-python/pull/45

Thanks! Works well on my end with a USB-6259.


[FLINK-21700][security] Add an option to disable credential retrieval on a secure cluster

Linked to ISSUE.
Why
I want to support Flink Batch Action on Oozie.
As we know, Oozie will obtain HDFS/HBase delegation token using Hadoop proxy user mechanism before starting Flink submitter cli. If not, Flink will obtain delegation token again, this will cause exception.
Actually, Spark support disable fetching delegation token on Spark client, related Spark doc.
So i think Flink should allow to disable fetching Hadoop delegation token on Yarn.

Hi @lirui-apache , @XComp , could you help to review this PR?
Thanks

Thank for the PR. I'm still trying to understand this change. Left some minor comments. Also, you should change the commit message/title following the format "[FLINK-21700][yarn] xxxx".

Ok. i will force push it to correct commit log later.

@zuston Could you squash your PR into one commit? Also the CI failed because of the style check, you could execute mvn com.diffplug.spotless:spotless-maven-plugin:apply to resolve it.

@zuston Could you squash your PR into one commit? Also the CI failed because of the style check, you could execute mvn com.diffplug.spotless:spotless-maven-plugin:apply to resolve it.

Done. Thanks

Hi @zuston and thanks for your contribution.
I'm wondering whether we can test that Flink is behaving in the right way when disabling the newly introduced flag. Right now, no test is covering this specific behavior.

It only works when credentials existed in UGI and no keytab is specified. I think It's hard to test, could you give me some ideas on it? @XComp

Hi @zuston and thanks for your contribution.
I'm wondering whether we can test that Flink is behaving in the right way when disabling the newly introduced flag. Right now, no test is covering this specific behavior.

It only works when credentials existed in UGI and no keytab is specified. I think It's hard to test, could you give me some ideas on it? @XComp

That's a good questions. @rmetzger How did we deal with these kind of problems in the past? Was it sufficient enough to have a manual test with the external system to see that it worked properly?

Do i need to submit another PR about changing identify to service in credentials?
Actually there  is no relationship between this PR and identify changes.
@XComp

Do i need to submit another PR about changing identify to service in credentials?
Actually there is no relationship between this PR and identify changes.
@XComp

Putting it in a separate commit within this PR is good enough if we go with this change.

Hi @zuston and thanks for your contribution.
I'm wondering whether we can test that Flink is behaving in the right way when disabling the newly introduced flag. Right now, no test is covering this specific behavior.

It only works when credentials existed in UGI and no keytab is specified. I think It's hard to test, could you give me some ideas on it? @XComp

That's a good questions. @rmetzger How did we deal with these kind of problems in the past? Was it sufficient enough to have a manual test with the external system to see that it worked properly?

Could you come up with a manual test which is documented properly in the PR to make it reproducible?

Thanks @XComp.
Ok, i will attach a detailed manual test in this PR later.

Update commits. Please check it. @XComp @lirui-apache @wangyang0918 @KarmaGYZ

@XComp Thanks for your reply

I'm still not comfortable with this change considering that it's not tested. Have you had the chance to check whether it works? Could you provide this manual test in a reproducible fashion (e.g. docker) or is this too much of an effort?

Sorry, I am not very familiar with the Flink code. Can you provide a test case link about with kerberos HDFS? I want to refer to it and add test case.
How to reproduce it?
I think you can fetch HDFS delegation token before submitting Flink job on Yarn, it will throw exception.

Based on what I read about it, the issue is that Apache Oozie utilizes Apache Hadoop's ProxyUser which impersonates the actual user which has access to the actual data. I still don't understand why the delegation token fetching causes an error. Is it because the Flink job would be still submitted under the "normal" user instead of the Oozie user?

Not yet. Oozie will submit Flink job without keytab and only rely on delegation token to access HDFS.

The Yarn test cases are not that easy to maintain. I was just wondering whether you could provide an easy-to-run skeleton for reviewers to rerun the test you performed already. I am not familiar with Oozie/Yarn/Kerberos to set it up. Or would that be to tricky to accomplish?

@XComp Done

@XComp Thanks for your reply.
Docs is updated and latest commit is uploaded. And this commit has made the following changes compared to the last one.

Update the security.kerberos.fetch.delegation-token conf description
Directly use token.getService as credential alias, instead of creating a new object, like Text alias = new Text(token.getService).
Rename some variables.
Some minor changes

Thanks for the update @zuston . I'm just wondering why you squashed the two commits into one again. Considering that the two changes (service name as alias and new parameter) are independent of each other I would favor @wangyang0918 's proposal of keeping the two commits separate.

@XComp Thanks for your reply.
Done.

Gentle ping @XComp

@XComp Thanks~. All resolved

@XComp. So i need to remove the token alias commit? Right?
And i will submit new one to solve it.

@XComp Done.

@XComp Thanks for your review. All done.

@XComp Gently ping. Could you help merge it?

Sure, I will initiate it as soon as @wangyang0918 gave it another pass. He has it on his ToDo list and will do it soon'ish

@wangyang0918 All done. Thanks

Thanks for the contribution and the reviews. Merging.


Fix spec output

Hi,
We have seen a little issue inside bacon where if you make a call to view that doesn't exist inside a test, bacon library prints the error correctly in console but the results line at the end says 0 erros.
It seems that when an exception got triggered, if the exception was an Error class, it was incrementing the error counter wrong.
I've seen that the original bacon project had already issues with the output not printing well and they fix it (partially) here at line 10,
https://github.com/chneukirchen/bacon/commit/55ac89c3f25e7a24fa5306e400af76483d2144ce


Add Linters

Detailed Description
Add the same linters used in other OCF repos, and fix any issues
Context
Its good to keep the code style and such the same throughout our codebase
Possible Implementation
Any of the .github folders in the main OCF repos should work

This has been done for awhile now. Still need unit tests though


Firewall rules are inherited from master

The doc says "When you create a replica, it doesn't inherit the firewall rules or VNet service endpoint of the master server.", but after I tested with a newly created replica, it seems that it inherits firewall rules from the master and ignores firewall rules configured on the replica in the Azure portal. Which one is correct behavior?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f8660100-8879-6519-3db2-14039a3c5a17
Version Independent ID: 64c84402-86e5-ae99-a6b5-8de8d8388e5f
Content: Read replicas in Azure Database for PostgreSQL
Content Source: articles/postgresql/concepts-read-replicas.md
Service: postgresql
GitHub Login: @rachel-msft
Microsoft Alias: raagyema

Thank you for the detailed feedback.
Let me try this at my end and I will get back to you soon.

Hi @pjknkda. Can you confirm that you were connecting to the replica server? You can check this by running the query SELECT pg_is_in_recovery();
If that returns false you are connecting to the master server. You can cross-check with the servername in your connection string (both the username and servername fields).

Oh, I found the reason.
When I first tried, both servers respond to the same output 'f',
pjknkda@myserver:~/$ psql -h test-db.postgres.database.azure.com -U adminuser@test-db -d test -c 'SELECT pg_is_in_recovery();'
Password for user adminuser@test-db:
 pg_is_in_recovery
-------------------
 f
(1 row)

pjknkda@myserver:~/$ psql -h test-db-replica.postgres.database.azure.com -U adminuser@test-db -d test -c 'SELECT pg_is_in_recovery();'
Password for user adminuser@test-db:
 pg_is_in_recovery
-------------------
 f
(1 row)

Meanwhile, I realized that the server name in the user name (behind @ symbol) also need to be changed (I guess Azure infrastructure use that part to proxy incoming connections to the real server), and now the response is retruned from the real slave server with the configured firewall rules.
pjknkda@myserver:~/$ psql -h test-db-replica.postgres.database.azure.com -U adminuser@test-db-replica -d test -c 'SELECT pg_is_in_recovery();'
Password for user adminuser@test-db-replica:
 pg_is_in_recovery
-------------------
 t
(1 row)

Thank you for your advise!

Thanks for confirming.
#please-close


ProxyFix does not work behind an AWS ELB with healthchecks

The ProxyFix middleware requires an argument that defines the amount of proxies in front of the application.
If you run the setup AWS ELB > NGINX > For example flask you would want the argument num_proxies=1. However if you point the health check to the same URL it will not have the correct headers and report the instance as crashed (with an exception since the proxy is out of range).
I wrote my own ProxyFix for now, and I suppose this is somewhat of an edge case but I thought I'd report it just the same. :)

It just hit us straight in the eye! Since this looks like it's not fixed yet.. @moodh do you still have that fix lying around somewhere?


allow to use tls for kafka

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area config
/area outputs

/area tests

What this PR does / why we need it:
Which issue(s) this PR fixes:
#547

Fixes #
Special notes for your reviewer:

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Issif
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Issif]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Issif I was about to PR this same feature! https://github.com/ibice/falcosidekick/commit/7a3aa25fb8b3c0d774048c357f64ba83eee3d2f7
In my solution, I included support for a custom CA. I suppose you plan to support it in the future since you added it to types.go. I'd be happy to add it once this gets merged.
One question out of curiosity, what's the reason behind this? I understand there isn't any client certificate involved.
https://github.com/falcosecurity/falcosidekick/blob/a5ca9ec3575c3107876e4dd0d46790ae4a545584/outputs/kafka.go#L34

@Issif I was about to PR this same feature! ibice@7a3aa25
In my solution, I included support for a custom CA. I suppose you plan to support it in the future since you added it to types.go. I'd be happy to add it once this gets merged.
One question out of curiosity, what's the reason behind this? I understand there isn't any client certificate involved.
https://github.com/falcosecurity/falcosidekick/blob/a5ca9ec3575c3107876e4dd0d46790ae4a545584/outputs/kafka.go#L34

I removed the field for the custom ca, I think this config should be global for all outputs, like we do with the key/certs for mTLS. This is something we could add in 2.29.0, don't want to block the 2.28.0 for that.
For the RequireAndVerifyClientCert it's useless, I removed it, thanks for the notice.

@Issif ok, we can rely on the system CAs for now.
Configuring the CA file globally makes sense. Tell me if you need help, I'll be happy to contribute.
Thanks for the feature!

Your contribution is welcome, you can already work on the subject, I'll just add the 2.29.0 as milestone.

LGTM label has been added.  Git tree hash: 50a4379ecfcdd650787d277471a69f7fc3ba67a4


Add additional reference test for determinism

Proposal for a test to verify deterministic behavior when reporting a list of evaluations at once (here as initial data), even if the order of the list varies.
Currently all optimizer pass the test (EDIT: except BOHB), but still I'm not 100% sure: is this always the desired behavior? Or are there cases, where we expect and desire different optimizer behavior depending on the order?
Signed-off-by: Buech Holger<EMAIL_ADDRESS>

Note: We might want to integrate the fix for failing BOHB in the same PR.

@sfalkner: We can try to solve this is this PR, or do that in a separate one, if we can tolerate a skipped tests :)

I already proposed a new function to solve this in #65. I'm going to rebase this branch here on top of that and try to apply the new function on BOHB to make it pass the test.


Support comments

Support this grunt plugin the comments option as well?
https://github.com/css/csso#command-line

Great idea! Wanna send a PR ? :1st_place_medal:


feat: improved app header style

Proposed Changes

Cleaner style for App header.
Improved margins for page title text.
Better style for background graphic

Adding that image was a mistake. I had the size down to 180kb. Will fix.

Size reduced to 100k.


Why is my file archived by you guys on Archive.org without my permission or consent?

Ref: https://web.archive.org/web/20230723001322/https://objects.githubusercontent.com/github-production-release-asset-2e65be/353324441/5edd7e80-9911-11eb-8efc-5b9a53a78272?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230723%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230723T001321Z&X-Amz-Expires=300&X-Amz-Signature=924632f158966f0190dc739026baa3d40c488676d4e3d652772f461fa818950c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=353324441&response-content-disposition=attachment%3B filename%3Ddreamingoflilianfont.ttf&response-content-type=application%2Foctet-stream
Hi @textfiles @archiveteamdockerhub @ArchiveTeam,
My name is Warren Woodhouse, font designer and distributor, who acquired the Ænigmate Productions library of fonts back in 2010.
I’ve noticed that you guys have crawled my font files without my permission or consent. As I have my own account on The Internet Archive and my content is currently being backed up on there as well as on GitHub, it’s advised that you don’t crawl my contents again unless you request to do so beforehand (you may use the issues section in the repository of my fonts repo or use my contact form on my Tumblr blog). Doing so should only be a last resort (for example, if my site was shutting down). In this case, my site was still active before I moved to Tumblr. My contents are still up on GitHub and on The Internet Archive (except for my font files as I’m still working on uploading them to there).
You’ll notice I’ve included the crawl link that you’ve generated on The Internet Archive. You’ll need to not crawl my archives and files again unless you’ve requested permission. This eliminates the problem of potential spammers and also eliminates the threat of potential copyright infringement.
This is just to inform you rather than punish you for your actions. Archiving is a glorious pastime but also comes with a landscape of legal issues, which I’m just making sure you’re aware of my own rights regarding created contents (license agreement and terms are available on my Tumblr blog).
You can find the full archive of my fonts (minus forks, pulls and crawls) available at https://github.com/warrenwoodhouse/fonts/releases
You can find my Font License Agreement at https://warrenwoodhouse.tumblr.com/terms
Thanks for taking the time to read this message and I look forward to reading your reply soon.
Regards,
Warren Woodhouse

Hello, I believe you want @.***
On Mon, Jul 31, 2023, 5:31 AM Warren Woodhouse @.***>
wrote:

Ref:
https://web.archive.org/web/20230723001322/https://objects.githubusercontent.com/github-production-release-asset-2e65be/353324441/5edd7e80-9911-11eb-8efc-5b9a53a78272?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230723%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230723T001321Z&X-Amz-Expires=300&X-Amz-Signature=924632f158966f0190dc739026baa3d40c488676d4e3d652772f461fa818950c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=353324441&response-content-disposition=attachment%3B filename%3Ddreamingoflilianfont.ttf&response-content-type=application%2Foctet-stream
Hi @textfiles https://github.com/textfiles @archiveteamdockerhub
https://github.com/archiveteamdockerhub @ArchiveTeam
https://github.com/ArchiveTeam,
My name is Warren Woodhouse, font designer and distributor, who acquired
the Ænigmate Productions library of fonts back in 2010.
I’ve noticed that you guys have crawled my font files without my
permission or consent. As I have my own account on The Internet Archive and
my content is currently being backed up on there as well as on GitHub, it’s
advised that you don’t crawl my contents again unless you request to do so
beforehand (you may use the issues section in the repository of my fonts
repo or use my contact form on my Tumblr blog). Doing so should only be a
last resort (for example, if my site was shutting down). In this case, my
site was still active before I moved to Tumblr. My contents are still up on
GitHub and on The Internet Archive (except for my font files as I’m still
working on uploading them to there).
You’ll notice I’ve included the crawl link that you’ve generated on The
Internet Archive. You’ll need to not crawl my archives and files again
unless you’ve requested permission. This eliminates the problem of
potential spammers and also eliminates the threat of potential copyright
infringement.
This is just to inform you rather than punish you for your actions.
Archiving is a glorious pastime but also comes with a landscape of legal
issues, which I’m just making sure you’re aware of my own rights regarding
created contents (license agreement and terms are available on my Tumblr
blog).
You can find the full archive of my fonts (minus forks, pulls and crawls)
available at https://github.com/warrenwoodhouse/fonts/releases
You can find my Font License Agreement at
https://warrenwoodhouse.tumblr.com/terms
Thanks for taking the time to read this message and I look forward to
reading your reply soon.
Regards,
Warren Woodhouse
—
Reply to this email directly, view it on GitHub
https://github.com/ArchiveTeam/ArchiveBot/issues/561, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAMTTTFDX543FPPPZHDZJH3XS53PBANCNFSM6AAAAAA257H53M
.
You are receiving this because you were mentioned.Message ID:
@.***>

I believe I want … all I see is asterisk symbols! 😂😂 You can email the link at https://warrenwoodhouse.tumblr.com/ask , if this helps.

No, I see the same thing. GitHub apparently redacts email addresses when they appear in comments.

I actually never used the comments feature before. Interestingly, GitHub doesn’t redact email addresses in the readme files or the other .md files.

Cheers. I think I might have a filter applied somewhere. Not sure where. Anyways, I’ll get in touch with them about it. Anyways, on a related note, web crawlers are useful tools. I’ve used Internet Archive for archiving my old websites for years (back in 2009 to present). Most of my archived stuff is on GitHub and Internet Archive. I actually used to use Google Code Projects before they closed it down. I managed to move my stuff from there to GitHub in like a day.

Why do you care so much about your fonts out of all things? I know that you have a legal right to your property, but I mean come on, this is as petty as Nintendo continuing to take down emulation of their games that were created decades ago.

Protecting work isn’t petty. In the case of Nintendo, it’s how it’s handled
correctly when archiving. I’ve decided to not pursue you guys but this is
more of a notice of not to archive my work unless you request beforehand,
so that you understand my own rights as is. Archiving is a noble and
fragile ground at the best of times. It’s rather, that I hope to protect my
work by archiving it myself or requesting that others archive it on my
behalf (have done on a number of occasions). Nintendo taking down emulation
games is a completely different matter altogether. Depending on how old the
software is, under legal law, anything other 60 to 80 years is considered
to be in the public domain. My work is not in the public domain.
Can understand where you guys are concerned though. You can consider this
issue resolved but as I’ve stated earlier, my work is for me to archive
unless I’ve stated otherwise. I’m actually using my GitHub and Internet
Archive accounts for archiving my work, so it’s technically an archive
anyway.
On Saturday, 2 September 2023, syndiate @.***> wrote:

Why do you care so much about your fonts out of all things? I know that
you have a legal right to your property, but I mean come on, this is as
petty as Nintendo continuing to take down emulation of their games that
were created decades ago.
—
Reply to this email directly, view it on GitHub
https://github.com/ArchiveTeam/ArchiveBot/issues/561#issuecomment-1703547970,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB4CMTLWTU4E3N7KQXAV5Y3XYJ3MHANCNFSM6AAAAAA257H53M
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
Warren Woodhouse
Gamer, Guitarist, Musician, Artist, Graphic Designer, Typographer,
Photographer, Blogger & YouTuber

STAY IN TOUCH
• Official Website: https://warrenwoodhouse.tumblr.com/
• Official Blog: https://warrenwoodhouse.tumblr.com/
• Photos: https://warrenwoodhouse.tumblr.com/tagged/photos
• Artwork: https://warrenwoodhouse.tumblr.com/tagged/artwork
• YouTube: https://youtube.com/user/warrenwoodhouse
• YouTube Gaming: https://gaming.youtube.com/user/warrenwoodhouse
• Twitter / X: https://twitter.com/warrenwoodhouse
• Pinterest: https://pinterest.com/worldofpins
• Contact me on Facebook Messenger:
https://messenger.com/t/warrenwoodhousesite
• PlayStation Profile: warren-woodhouse
• XBOX Gamertag: WarrenWoodhouse
• Podcast:
https://warrenwoodhouse.tumblr.com/tagged/podcasts-warrenwoodhouse
• GitHub: https://github.com/warrenwoodhouse

I didn't recognize that this was a github discussion. I'll say that in the future, or now, the best way to contact about Wayback redactions and issues is to mail<EMAIL_ADDRESS>which is the e-mail I was trying to get across before.

I know, you mentioned it before. I’ll bare it in mind incase of any future
incidents. Thanks again for letting me know though.
On Sunday, 3 September 2023, Jason Scott @.***> wrote:

I didn't recognize that this was a github discussion. I'll say that in the
future, or now, the best way to contact about Wayback redactions and issues
is to mail @.***, which is the e-mail I was trying to get
across before.
—
Reply to this email directly, view it on GitHub
https://github.com/ArchiveTeam/ArchiveBot/issues/561#issuecomment-1703961220,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB4CMTMHKAEEPGCG2WTQFFTXYO4RHANCNFSM6AAAAAA257H53M
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
Warren Woodhouse
Gamer, Guitarist, Musician, Artist, Graphic Designer, Typographer,
Photographer, Blogger & YouTuber

STAY IN TOUCH
• Official Website: https://warrenwoodhouse.tumblr.com/
• Official Blog: https://warrenwoodhouse.tumblr.com/
• Photos: https://warrenwoodhouse.tumblr.com/tagged/photos
• Artwork: https://warrenwoodhouse.tumblr.com/tagged/artwork
• YouTube: https://youtube.com/user/warrenwoodhouse
• YouTube Gaming: https://gaming.youtube.com/user/warrenwoodhouse
• Twitter / X: https://twitter.com/warrenwoodhouse
• Pinterest: https://pinterest.com/worldofpins
• Contact me on Facebook Messenger:
https://messenger.com/t/warrenwoodhousesite
• PlayStation Profile: warren-woodhouse
• XBOX Gamertag: WarrenWoodhouse
• Podcast:
https://warrenwoodhouse.tumblr.com/tagged/podcasts-warrenwoodhouse
• GitHub: https://github.com/warrenwoodhouse


CoverageException: Can't add file tracer data for unmeasured file

A combination of django, pytest, pytest-cov, django-coverage-plugin and pytest-django seems to interact with each other causing CoverageException when tests execute code that renders templates. My guess is it's somewhere in django-coverage-plugin as disabling it, removes the exception.
I created an example Django project that exhibits this issue: https://github.com/suda/pytest_django_coverage_test
Not sure if it's a bug or misconfiguration but I hope this repo will be useful for figuring out the root cause 😊

I've also run into this problem 👍

An interesting bug report, thanks!
There are a few things going on here.  If I add the -s switch to pytest, then I can see the first issue:
render_something/tests.py Coverage.py warning: Disabling plug-in 'django_coverage_plugin.DjangoTemplatePlugin' due to an exception:
Traceback (most recent call last):
  File "/usr/local/virtualenvs/tmp-91b59576424c64de/lib/python3.7/site-packages/coverage/inorout.py", line 263, in should_trace
    file_tracer = plugin.file_tracer(canonical)
  File "/usr/local/virtualenvs/tmp-91b59576424c64de/lib/python3.7/site-packages/django_coverage_plugin/plugin.py", line 179, in file_tracer
    self.debug_checked = check_debug()
  File "/usr/local/virtualenvs/tmp-91b59576424c64de/lib/python3.7/site-packages/django_coverage_plugin/plugin.py", line 86, in check_debug
    "Template debugging must be enabled in settings."
django_coverage_plugin.plugin.DjangoTemplatePluginException: Template debugging must be enabled in settings.

Pytest normally captures all the output, so you couldn't see this message.  The plugin is being disabled, but later, coverage doesn't take that into account, and tries to do some final bookkeeping for a plugin which never recorded any lines.
You can fix the problem by enabling debugging in your template engine:
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'debug': True,       ## <--- add this line.
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

I'll have to think about how to fix coverage so that it properly tracks disabled plugins.  I've written it up as https://github.com/nedbat/coveragepy/issues/1011.


Boos 4138 llm agent refining

Summary
Improvement of LLM agent reasoning and action execution including a refactoring of the llm agent code and prompts.
In general, these changes make the LLM agent capable of taking actions over 256 steps without any grammar errors or general code errors. The pudding in all of this is that the LLM agent is now taking appropriate and effective actions much more often than previously.
I'm currently running a benchmark of this on 256 steps to get us a final reward score to take note of :)
Major Changes

Two hop agents, reason around observation space and select a node to act on defensively (or select none for no action)
Given any selected node, decide what action to take on it with respect to the reasoning output from the previous hop.
Restructure of LLM class methods, including the grammar json to regex hack (for output order preservation) and a 'new_literals' which allows a property to be defined as an ENUM after the pydantic model has been defined in the code (used in the node selection).
All prompt structures changes to be simpler to work with

Minor changes

Integrate changes into streamlit app
Changed info attribute of EnvironmentState class to 'prompt' and add 'reasoning' attribute.
Only showing the LLM the last 20 observation space changes as to not clogg the prompt too much and reduce risk of max token limit errors (would still happen if the network was massive, so in future we'll need to address this if it becomes a problemo)

@Johnno1011 hey sorry i just merged jack's pr, could you solve the merge conflicts pwss tyy


Useless hint with optional package

$ jbuilder external-lib-deps --missing --only-packages lwt @install
Error: The following required libraries are missing in the default context:
- camlp4 (optional)
Hint: try: opam install

The hint should provide a better message.

Indeed, thanks for the report


Document CCM upgrade process recommendations

A customer recently asked about upgrading from an older version of the CCM (0.1.0 => 0.4.0). We should add this to documentation as it's something that's likely to come up fairly often.

$ export RELEASE=0.4.0
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-cloud-controller-manager-rbac.yaml
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-cloud-controller-manager.yaml


fix: allow for type apply after splice

resolves: https://github.com/scalameta/metals/issues/6483

nice and beautiful this time, thank you!

Initially I wanted it to also cover:
def fn[A](a: A) = ???
{
  [B] => (c: B) => fn[B](c)
}[Int](0)

Since this snippet compiles but I don't think it's ever used and worth the hassle.


Operation not permitted

$ ./setup_blockchainNetwork_v2.sh
...
Generating the required artifacts for Blockchain network
Running: kubectl create -f /Users/kylehz/Downloads/blockchain-network-on-kubernetes/configFiles/generateArtifactsJob.yaml
job.batch/utils created
Waiting for generateArtifacts job to complete
...
Waiting for generateArtifacts job to complete
There is an error in utils job. Please check logs.

$ kubectl get jobs --watch
NAME            COMPLETIONS   DURATION   AGE
copyartifacts   1/1           32s        9m34s
utils           0/1           9m1s       9m1s

$ kubectl get pods | grep utils
utils-95mjv                   1/2     Error       0          9m55s

$ kubectl logs utils-95mjv -c cryptogen
<EMAIL_ADDRESS><EMAIL_ADDRESS>find: ‘/shared/Proxifier-Euz62I’: Permission denied
chmod: changing permissions of '/shared': Operation not permitted
chmod: changing permissions of '/shared/Proxifier-Euz62I': Operation not permitted

$ kubectl version --short
Client Version: v1.15.5
Server Version: v1.17.3

@kylehz , I'll check with this kubectl version and update you. Thanks.

@kylehz - I did not face this issue when I tried with Kubernetes Service hosted on IBM Cloud. Still you can take the latest code and retry. I have updated the code repository for the Kubernetes 1.16. Let me know if it works for you Thanks.

@maheshwarishikha - I got the same issue.
$ k logs utils-9jrd6 cryptogen
/shared/crypto-config/peerOrganizations/org2.example.com/peers/peer1.org2.example.com/msp/keystore/36235e839138677c476cb1d6b51b7ccf9b8c068db1529a1057f15dd81f54d217_sk
/shared/crypto-config/peerOrganizations/org2.example.com/peers/peer1.org2.example.com/msp/keystore
chmod: changing permissions of '/shared': Operation not permitted
$ kubectl version --short
Client Version: v1.15.5
Server Version: v1.15.5
I run ./setup_blockchainNetwork_v2.sh and i have enabled kubernetes in docker-desktop locally.

@kylehz - hope it worked for you. Closing the issue because of inactivity. Feel free to reopen or open a new issue if required. Thanks.


Import MSFAA information from SFAS

MUST CONFIRM WITH DEVS IF THIS IS STILL REQUIRED
Describe the task
A clear and concise description of what the task is.
Acceptance Criteria

[ ] first
[ ] second
[ ] third

Additional context

Add any other context about the task here.
Or here

Closing this ticket as obsolete - MSFAA data is available.


Create Prometheus.md

Adds point in docs on how to integrate Prometheus alerts in Rocket Chat with https://github.com/pavel-kazhavets/AlertmanagerRocketChat

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

This should be a very useful addition!  Thanks for contributing

Good to know. Thank you!


Change all instances of ripple to use the TSX version

Once all components have been merged into the branch feat/typescript, we can update all components using react-ripple, to depend on the tsx file.
For example in react-button is importing react-ripple like so:
// TODO: fix with
// @ts-ignore
import withRipple from '@material/react-ripple';

fixed with https://github.com/material-components/material-components-web-react/pull/566


Use kibana.version correctly

For the kibana.version setting to work correctly, it needs to be a version property in the kibana object.
Updated the package.json file here to work correctly. Probably a good idea to backport this to any other branched you're continuing to support. Cheers!

Thanks. Will backport to required branches.


Add tests covering multi-unit prometheus deployments

Issue
Currently there are no tests covering multi-unit prometheus deployments.
Closes #173.
Solution
Add tests.
Context
NTA.
Testing Instructions

tox -e integration -- -k test_prometheus_scrape_multiunit

Release Notes
Add tests covering multi-unit prometheus deployments.

@rbarry82 mind taking a look at what I have so far?

The changes here are broadly good, though I personally think "is the Prometheus endpoint available?" isn't a really sufficient test.
I guess the "ideal" would probably be something like:

Deploy a single Prometheus and tester. Verify we have metrics.
Scale Prometheus up to 3 units. Verify they all have metrics (this could just be a further condition on the previous test).
 Verify all 3 are firing. (could also be part of the previous test)
Upgrade Prometheus. Without waiting for idle, also scale up the tester so a relation-joined fires while it's in the middle of doing things, or start an upgrade of the tester first with different alert rules to make sure that Prometheus behaves "as expected" when there's a collision between relation events and upgrade events.
Make sure metrics keep continuity and alerts which were firing are still firing (could be additional assertions in the scaling test)
Scale the tester up and make sure the new tester's metrics show up

Scaling down is much harder to test since we don't necessarily know "which" tester is going to go away, but this way we exercise the functionality of Prometheus rather than just "does it exist"?

"is the Prometheus endpoint available?" isn't a really sufficient test.

Agreed. What I currently have under # THEN nothing breaks is mostly a placeholder and we should add more.

 Verify all 3 are firing.

It seems wrong to me to include this here, because we want to stay away from all-in-one tests:

Prometheus firing an alert is a "prometheus thing" that is tested in a different itest file.
As long as all the configs are the same for all units (which IS tested here), then I think we can safely assume they all behave the same.

Upgrade Prometheus. Without waiting for idle, also scale up the tester so a relation-joined fires while it's in the middle of doing things, or start an upgrade of the tester first with different alert rules to make sure that Prometheus behaves "as expected" when there's a collision between relation events and upgrade events.
Make sure metrics keep continuity and alerts which were firing are still firing (could be additional assertions in the scaling test)
Scale the tester up and make sure the new tester's metrics show up

Sounds like a good tests but I'm not sure if it's better to have them here or in test_upgrade_charm.

It seems wrong to me to include this here, because we want to stay away from all-in-one tests:

Prometheus firing an alert is a "prometheus thing" that is already tested in a different itest file.
As long as all the configs are the same for all units (which IS tested here), then I think we can safely assume they all behave the same.

I think that was sort of the discussion earlier about an e2e test just for each test, without adding in things from the bundle.
We test each piece in relative isolation, but it's like testing the brakes/steering linkage/drivetrain of a car without testing to make sure it actually drives. We don't need test_... for each test, but for each "major" step, because we have already been bitten by "safely assume" elsewhere.

Upgrade Prometheus. Without waiting for idle, also scale up the tester so a relation-joined fires while it's in the middle of doing things, or start an upgrade of the tester first with different alert rules to make sure that Prometheus behaves "as expected" when there's a collision between relation events and upgrade events.
Make sure metrics keep continuity and alerts which were firing are still firing (could be additional assertions in the scaling test)
Scale the tester up and make sure the new tester's metrics show up

Sounds like a good tests but I'm not sure if it's better to have them here or in test_upgrade_charm.

Since test_upgrade_charm doesn't deal with scaling or possibly async/staggered relation events if not all of the upgrades complete at the same time (and they probably won't). All other extant tests aren't trying to scale and upgrade and make sure that we don't have a break in upgrades, and, for better or worse, this is a very viable "real world" case if script refreshes a model while once it's verified in dev/preprod/whatever.

To verify metrics continuity, I'm thinking:
numSeries and numLabelPairs remains the exact same across a prom upgrade.
curl -s <IP_ADDRESS>:9090/api/v1/status/tsdb | jq ".data.headStats"
{
  "numSeries": 677,
  "numLabelPairs": 378,
  "chunkCount": 5950,
  "minTime":<PHONE_NUMBER>704,
  "maxTime":<PHONE_NUMBER>883
}

WDYT @rbarry82?

To verify metrics continuity, I'm thinking: numSeries and numLabelPairs remain the exact same across a prom upgrade.
curl -s <IP_ADDRESS>:9090/api/v1/status/tsdb | jq ".data.headStats"
{
  "numSeries": 677,
  "numLabelPairs": 378,
  "chunkCount": 5950,
  "minTime":<PHONE_NUMBER>704,
  "maxTime":<PHONE_NUMBER>883
}

WDYT @rbarry82?

Surprisingly these are not as stable as I hoped: sometimes two prom units have different values, and sometimes the values before and after the upgrade differ.
  File "/home/ubuntu/code2/prometheus-k8s-operator/tests/integration/test_prometheus_scrape_multiunit.py",
  line 157, in test_upgrade_prometheus
    assert head_stats_before == head_stats_after
AssertionError: assert {Stats(numSeries=499, numLabelPairs=352)} ==
                       {Stats(numSeries=499, numLabelPairs=352), Stats(numSeries=527, numLabelPairs=358)}
  Extra items in the right set:
  Stats(numSeries=527, numLabelPairs=358)

  File "/home/ubuntu/code/prometheus-k8s-operator/tests/integration/test_prometheus_scrape_multiunit.py",
  line 157, in test_upgrade_prometheus
    assert head_stats_before == head_stats_after
AssertionError: assert {Stats(numSeries=507, numLabelPairs=356)} == {Stats(numSeries=535, numLabelPairs=362)}

Ideas?

To verify metrics continuity, I'm thinking: numSeries and numLabelPairs remain the exact same across a prom upgrade.
curl -s <IP_ADDRESS>:9090/api/v1/status/tsdb | jq ".data.headStats"
{
  "numSeries": 677,
  "numLabelPairs": 378,
  "chunkCount": 5950,
  "minTime":<PHONE_NUMBER>704,
  "maxTime":<PHONE_NUMBER>883
}

WDYT @rbarry82?

Surprisingly these are not as stable as I hoped: sometimes two prom units have different values, and sometimes the values before and after the upgrade differ.
  File "/home/ubuntu/code2/prometheus-k8s-operator/tests/integration/test_prometheus_scrape_multiunit.py",
  line 157, in test_upgrade_prometheus
    assert head_stats_before == head_stats_after
AssertionError: assert {Stats(numSeries=499, numLabelPairs=352)} ==
                       {Stats(numSeries=499, numLabelPairs=352), Stats(numSeries=527, numLabelPairs=358)}
  Extra items in the right set:
  Stats(numSeries=527, numLabelPairs=358)

 File "/home/ubuntu/code/prometheus-k8s-operator/tests/integration/test_prometheus_scrape_multiunit.py",
 line 157, in test_upgrade_prometheus
   assert head_stats_before == head_stats_after
AssertionError: assert {Stats(numSeries=507, numLabelPairs=356)} == {Stats(numSeries=535, numLabelPairs=362)}

Ideas?

I would pretty much always expect the stats after to be greater, so that's a fair assertion.
It's more concerning that they differ. Guess would be that head_stats_before and head_stats_after should be lists, not set (or be dict with each unit as the key), and "extra items in the right set" is because the units diverged when one upgraded/came up faster, so the sets didn't ahve the same values.


Potentially faulty test case

reported: https://www.facebook.com/groups/free.code.camp.bengaluru/761870870602896/?ref=notif&notif_t=group_activity
Screen shot: https://scontent.fsnc1-1.fna.fbcdn.net/hphotos-xpf1/t31.0-8/11888635_963123300399960_4127102090052504172_o.jpg

I'm not sure, but it looks like the code looks for spaces and capitalizes the character at the next index? I'm pretty sure that the sHoRt AnD sToUt test case is the only one that fails because it's the only one where the supplied string starts with a lower case letter, and since the first character isn't preceeded by a space it's ignored by the campers code.

Hey it looks like this is a duplicate of #2424.
Closing this.

@benschenker I don't think this is a duplicate of that issue actually. If you read the camper's code, you'll see that they understood the need to lower-case everything that's not the first character, they just have a bug in their code. This issue should be closed, and the camper should be redirected to the RSAP wiki page.


Client missing description while EF Client has it.

The model in Ids4 Client.cs   does not appear to contain a description property of the client.  However the entity framework model Client.cs does contain a field called description.

public string Description { get; set; }

I am a bit new to Entity framework so not really sure if this is a bug or if I am doing something wrong.   Shouldn't the client object in Ids4 contain all the properties available in the database from EF?  or am I missing something.

I am on it only took me five minutes to add it.   Just trying to wrap my head around your test application in here so that i can force it to come with the consent screen with the added description on it.

Thanks for this -- I think we didn't understand where you had meant, which is mostly in the UI.

I finally had a real look at the PR. Basically you're looking for the description to be used in IS somehow... the description in the EF layer is meant for any custom admin UI. not necessarily for IS. I guess the only interesting place would be on the consent screen. Is that the main driver for this issue request?

done

yup our third parties need description on the consent screen.
thanks

ok, then perhaps we can try to re-apply the PR to the new structure. of you can always just do it in your UI.

the problem is simple the field exists in ef it doesn't not exist in the ids datamodel.  why exactly would the datamodel not include all values available in ef.   if you don't want description why was it added in the ef?
I can't just add it to the ui it doesn't exist.   I applied that pr three months ago because we needed it in production asap I told management it would be in the next release as I personally submitted a pr.
are we talking about waiting another three months for you to review and reject another one of my prs?   I am on vacation for another two weeks and won't be able to submit anything anyway.

We're actively working on 2.3 given that we have time now. 3 months ago we were't. If you're on holiday, I'd not bother and enjoy the time off instead.