System primitives

As a developer, I would like access to system primitives so I can use patterns from Mineral in other frameworks.
Assuming the research in #639 is fruitful, these could be tackled next, in order:

Create CSS & HTML primitives
Create documentation
Distribute CSS & HTML primitives
Investigate an automated approach to consider

What NOT to do:

Anything that would remove Glamorous
Epic refactors
Significant increases to build process
Manual solutions (should be another story)

This issue has been updated. See next comment for details.
Original content

As a developer, I would like access to system primitives so I can use patterns from Mineral in other frameworks.
Assuming the research in #639 is fruitful, these could be tackled next, in order:

Create CSS & HTML primitives
Create documentation
Distribute CSS & HTML primitives
Investigate an automated approach to consider

What NOT to do:

Anything that would remove Glamorous
Epic refactors
Significant increases to build process
Manual solutions (should be another story)

Due to the following reasons, we're closing this issue.
Primitives as a concept can be broken down into HTML and CSS.

HTML

We could potentially consume the HTML primitives within our React components. But we could find no examples of this in the wild, and, after some research, it seems like it would be very difficult, if not impossible.

The examples we did find Lightning's React and Blueprint simply write the HTML in their components separate from the provided HTML primitives.

Less ideal, we could potentially generate the HTML from our React components. This seems doable, at least. The HTML tags and attributes (aside from class) can be generated relatively easily, but that class attribute and the associated styles presents many challenges, described below.
Finally, we could follow Lightning React's and Blueprint's example and manually maintain a separate set of HTML primatives, creating two sources of truth, but manual solutions are a separate issue. This approach also still has most of the challenges with the CSS, described below.

CSS

Similar to HTML, we could potentially consume CSS primatives in our components. But doing so would require removing glamorous and is thus out of this issue's scope.
We could potentially generate CSS from our existing styles. Because we use glamorous, the classes on the HTML elements would be auto-generated nonsense. Converting these to be more helpful is likely possible, but may not have much value, because

glamorous produces a new class for each set of styles, e.g. a standard Button and a primary Button would have all the same styles except for color & background-color.
But we would want to provide CSS that uses a base + modifiers model, e.g. a standard Button would have most of the styles and a primary Button would have a modifier class that only changed the color & background-color.
Mapping from one to the other seems quite difficult, if even possible. This is true primarily for the reason demonstrated above, as well as many others, e.g. rtl support, our use of getNormalizedValue, etc...
If we don't do that mapping, then the static documented class would imply static styles, which isn't the case.

Finally, we could manually extract the CSS from our styles (predefined as a separate issue, and not all that straightforward)


Performance Issues on V 4.2

I noticed a degrade in performance going from Version 4.1 to Version 4.2
(No WebP enabled - Querystring Cache breaking enabled in cloudfront)
V 4.1:
https://d147ewtskziry6.cloudfront.net/eyJidWNrZXQiOiJuYW5vY2hpcC1wZXQtcGljdHVyZXMtZGV2Iiwia2V5IjoiUGV0UGljXzEyMzMyMTEyMzMucG5nIiwiY2FjaGUiOiIxNTg2NjE5MjM1IiwiZWRpdHMiOnsicm90YXRlIjpudWxsfX0=?some_random_number
V 4.2:
https://d1bpkp55zv2eya.cloudfront.net/eyJidWNrZXQiOiJuYW5vY2hpcC1wZXQtcGljdHVyZXMtZGV2Iiwia2V5IjoiUGV0UGljXzEyMzMyMTEyMzMucG5nIiwiY2FjaGUiOiIxNTg2NjE5MjM1IiwiZWRpdHMiOnsicm90YXRlIjpudWxsfX0=?some_random_number
(For verification replace some_random_number with a timestamp or something random
For V 4.1 I am getting around 2.0 - 2.5 seconds
For V 4.2 I am getting around 3.5 - 4.0 seconds
Both go against the same bucket - all in region us-east-1

@mberneis Thanks for your observation. I'll take a look at what would cause more latency on v4.2.

Thanks for waiting, @mberneis
I figured out that the performance is because of changing output format to webp. Currently, v4.2 has a broken source code to enable automatic webp, so every file is converted to webp no matter what parameter you provided.
I found that converting the output format to webp takes a long time: around 2s. As a result, overall performance has been degraded. We will fix this one in the next release.
Once again, thanks for your investigation.

@mberneis
Thanks for waiting. This one is fixed in v5.0.0.


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_075tcmcqsk/clone/recipe/meta.yaml:16: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_075tcmcqsk/clone/recipe/meta.yaml:22: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_075tcmcqsk/clone/recipe/meta.yaml:32: missing_pip_check: For pypi packages, pip check should be present in the test commands
Errors were found


I Can't use in sandbox environment

The Plugin fails to initialize for sandbox environment.
Could it be that logic in the validateChargeAndKey prevents the use of sandbox public keys because they start with pk_?

I was wrong! The plugin works on sandbox
Flutter clean fixed the issue.


Update DG to CS2101 & CS2103T standard

Updated DG Section 3.3 to 3.3.2  and Section 5.0 to 6.0. Removed Appendix A as it is not relavent.

I think you can just overwrite the older one, just use your file.

Resolved the merge conflict, merged.


Support resuming indexing

...and by extension also not replaying a WAL that has already been fully replayed.

I don't think this is actually valuable enough to do. We already stop indexing if we find a pre-existing index and don't have the force flag.

This is now needed for the live indexer.


Logging Review

Logs will become persistent and available to users even when a worker has been terminated. The following changes need to occur:

Fewer Info Logs
Move Info to Debug
Do not print out job info input or output
No multiline logs
Try and capture standard out better.

Logs have been cleaned.


Real-time processing of the BLE Advertisements

Since there were requests for the implementation of support for "switching" sensors (binary sensors like switches, the presence or motion sensors), I began to think about redesigning the platform for processing messages from sensors in real-time.
The lack of the current approach (periodic) I have already described in this release note:
Due to the fact that this component has a periodic nature of the operation, the state of the binary sensor will arrive with some delay. For example, if your period option is one minute (by default this is the case), and the state of the sensor has changed by 30 seconds of the period, then the state of the binary sensor entity in HA will change only after the remaining 30 seconds have passed.
Theoretically, we just need to transfer the parser code to the thread that receives messages from the interface ("HCIdump" thread). However, it is necessary to solve several problems arising in this case:

many sensors send quite a lot of messages, which leads to the issue of the minimum sufficient performance (the current approach eliminates this problem due to the fact that the collection of messages and their processing can occur "simultaneously" due to multithreading);
if the entity is updated for each received message, this will lead to a monstrous load on the storage system (database), therefore, it is necessary to provide the logic of accumulation and averaging of the readings of the measuring sensors;
since the component allows you to collect data from several interfaces, there is a problem of unjustified additional load due to the fact that often messages from all interfaces will be the same (it is necessary to transfer the duplicates processing logic to the earliest possible stage);

I am pleased to hear comments on this issue from all interested.
Is there any real reason to do this?

@Ernst79 what do you think about this issue?

I think, if you decide to change things, you should make a clear distinction between sensors and switches. Switches should react immediately. Sensors do not, and the period can already been set by the user. Binary sensors are the only exception, and kind of depend on the device they represent.
I could imagine that you make it  an option for binary sensors and switches, such that the user can choose wether he/she wants immediate response (with the warning of huge data flows if you do that for a sensor that sends data very often) or that he want to use it the normal/old way (which should be the default).
Which device(s) are you thinking of, that would need immediate response type of behavior?

We already have two devices in our requests: this is a repellent and a lamp with a motion sensor (there is no dump for it yet). I do not know such devices anymore (I have not studied the assortment). The Mija BLE standard also implies these types of binary objects: a leakage sensor, a door/window opening sensor, a gas sensor, a smoke sensor, and so on, I don’t remember exactly already.
Yes, there is no urgent need for such changes yet, but I want to be prepared for this )
Well, there is some "sporting" interest also ) I’ll experiment for now...

I created a new branch, realtime_tests, in which I will gradually move towards processing messages in real-time, as they arrive. At the moment, the BLE ADV parser is completely transferred to the HCIdump thread. While I do not see noticeable changes in the load on the system (rpi3b+, one bt-interface, 9 sensors, 5 of them with encryption). I’m watching further.

It seems that the realtime code is working fine. In any case, I do not see any problems so far. It was even possible to reduce the CPU load to about the level of a "periodic" code.
The only thing I do not have the opportunity to test the operation of binary sensors.
I’ll observe a week, and I will offer this code to try the owners of sensors with switches.
Branch realtime_tests, if anyone is interested.

I recently posted a 0.7.0-beta that implements real-time processing.
I do not see any problems with it, it works well for me. I’ll wait a little longer, and will merge it to the master branch.
The next step is to attempt to make component completely async without using additional threads (I try to follow component quality ladder)


Logo

Yeah, a dash reading a book IDk, this has to have a logo, please.

@renancaraujo you could help me with this 😬
haha


Expose CNI net directory in Helm chart?

Hi - thanks for all the work so far on Calico!
I'm using an immutable distribution that disallows writes to /etc/cni/net.d and was wondering if you'd be open to exposing that in the Helm chart values. I can write up a PR for it but thought I should ask first.
Expected Behavior

Current Behavior

Volumes for calico-node hard-coded to /etc/cni/net.d.
Possible Solution

Add property node.cni.netDir to values and default the mountPath in the Calico Helm chart to /etc/cni/net.d
Your Environment

Calico version: 3.72.2
Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.28.4
Operating System and version: NixOS 23.11
Link to your project (optional):

I think the first thing we'd need to do is add support for this in the tigera/operator repository.
We already have support for configuring this on Windows: https://github.com/tigera/operator/blob/master/api/v1/installation_types.go#L831-L845
We would need to add equivalent configuration options to the operator for Linux, and then these options will show up in the helm chart "for free" as a result (since we embed the Installation directly into values.yaml)


⚠ Fetchers has degraded performance

In acf3c09, Fetchers ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 346 ms

Resolved: Fetchers performance has improved in 714633a.


Broken pipe check not portable (i18n)

Hi,
Seems broken pipe error is tested without taking into account i18n so it is missed on most builds not using en:

(french example)
java.net.SocketException: Relais brisé (pipe)

Not sure the best fix (forcing the locale is likely not an option, whitelisting most of translation is maybe not)

@rmannibucau Thanks for bringing it to our attention. Maybe there is a way to force en locale just for the purposes of running the tests. What would be the benefit of testing under various locales?

@meltsufin Locale.setDefault done very early (way too early for maven or gradle) will set it globally but it will also impact other parts of the application (if jib is embed it breaks rest of the app, same for maven). So forcing the locale can be doable with some JVM internals abuse but I wouldn't go that way, it would stay too fragile and will not solve the issue and let JIB not fix the issue this test was introduced for. I guess a more valid solution would be to dig why there is a broken pipe (Expect: 100-continue? which is testable without checking the exception) and only if needed, extend the http client to throw a more specific exception on such a case.

If anyone can come up with a check that is not message-dependent, please make a contribution. Otherwise, I think we'll leave it as a known limitation.

The opposite can be worth checking, ie not handling it at all.
It is only about the error message and it is wrong regularly (whereas exception/cause does not lie) so maybe just drop the tentative to guess what happent?
The wrapping exception can point all potential cause instead of picking a random one.
Sounds like a good unification and solution to me.

@chanseokoh WDYT?

I think the status quo is fine. It doesn't do any handling/processing/unrolling on the exception; it doesn't change the behavior of Jib at all, except that it only puts a couple more console output lines on the console for the US locale. And this has been actually very useful, because the only message you get from the broken pipe SocketException is just the two words "broken pipe" and nothing else most of the time. Moreover, I believe the exception is almost all the time a terminal one without a wrapped exception.

@chanseokoh well fact is US locale is not the most common one even if common and when it fails with a broken pipe you can end up having a misleading error message (don't have it handy but think it was "check your credentials") which is worse.
side note: when i mentionned a wrapping exception it was about enriching the exception handling with a jib wrapping exception with more details about potential causes.
what is helping is to show the potential causes to let people check them, this is why I proposed to show them all (it is 2-3) in the error message and not try to be over clever until there is a way (which means enriching the http client to catch the connection error lower in the code layers).

a misleading error message (don't have it handy but think it was "check your credentials")

What we log (only in the case of the US locale, which is not the most common one, of course) is
broken pipe: the server shut down the connection. Check the server log if possible. This could also be a proxy issue. For example, a proxy may prevent sending packets that are too large.

which we believe is not misleading.

until there is a way (which means enriching the http client to catch the connection error lower in the code layers).

I don't think there is a way to reliably determine in Java at any layer (lower or higher) whether a socket exception is due to a broken pipe, until future Java versions introduce a dedicated exception class for that (which I think will not happen). The best effort is to check an exception message string, which can never be reliable, since future Java versions may change the string and messages are different on different locales.

I don't think there is a way to reliably determine in Java at any layer (lower or higher) whether a socket exception is due to a broken pipe, until future Java versions introduce a dedicated exception class for that (which I think will not happen).

Since you are not using java http client but apache http client which is at socket level I think it is quite doable to track the socket to know if the client or server cut the connection.
Overall if not reliable I wouldn't add the feature at all to have a reliable behavior whatever locale is used.

which we believe is not misleading.

it is because it happens when SSL is wrongly setup and or the server rejects it due to some overload pike (openshift there).
An interesting comment from the JDK can help to refine these cases if desired (./test/jdk/java/net/httpclient/CancelledResponse.java):
             // if SSL then we might get a "Broken Pipe", or a
             // RuntimeException wrapping an InvalidAlgorithmParameterException
             // (probably if the channel is closed during the handshake),
             // otherwise we get a "Socket closed".

So the isBrokenPipe method can need some refinement of the stack to be more accurate probably.

Thanks for your input, but I think we want to keep the current logging. It's just that the user additionally sees a bit more longer message "broken pipe: " than the two words "broken pipe" if in the US locale. Other than showing a bit more message, Jib doesn't do anything more than that; whether on US or not, Jib works exactly the same way, since there's no way to reliably determine if it's broken pipe or not.
And I still think the message is not misleading. "Broken pipe" means the server abruptly and unexpectedly closed the pipe for whatever reason while it shouldn't, whether intentionally or due to system overload, and the client (Jib) is only left uninformed and speculating. There's nothing much you can do from the client side. In the SSL case too, the gist is that if the handshake cannot be fulfilled, there is a chance that the client may see various kinds of errors including "broken pipe" depending on the situation and how the server reacts. But from the client side, it is still a broken pipe leading to ultimate symptom of SSL handshake. Something went wrong unexpectedly on the server side (or in the middle) and no fault on the client side. The nature of the error tells us that something to be fixed on the server side, which is what the current logging is trying to say.

I think it is quite doable to track the socket to know if the client or server cut the connection.

Not sure I follow "track the socket." In the broken pipe case, the client is abruptly left hanging without any preceding sign. The kernel just reports that the other end of a pipe is broken when a client attempts to write bytes to the pipe.

@chanseokoh well, don't want to fight a lot for that since it just misguide the user to the error cause (no proxy error or anything like that in the case I mentionned to be concrete and the different view between users just because they use a different locale does not help to solve the actual issue).
Yes the error is reported but the cause can be known adding more context in the http client, either SSL stage or before/after leading to a finer cause identification (I assume only 3-4 actual causes can be).
So options I see are:

replace the proxy message by a longer one listing SSL, proxy, server issue (load or not) for example
try to detect these cases (the SSL one can be identified in general but can require some HTTP client "instrumention")
(not satisfying but at least enable to work between people more easily) at least document how to force a locale to let people reproduce issues and potentially enable to disable these US only handling (false for maven plugin for example)

I like the option 1, which is what we intended with the current logging (i.e., providing more context and explanation for what a "broken pipe" can mean for the user). Contributions are welcome. I added "good first issue."

how to force a locale to let people reproduce issues and potentially enable to disable these US only handling

Hmm... I don't really get why one needs to disable it? Jib currently doesn't "handle" it, so whether US or not, the same issue and symptom will be reproducible. As I said, the current logic just adds a couple more lines to the existing error lines without interfering Jib's usual execution flow in any possible way.

Hmm... I don't really get why one needs to disable it? Jib currently doesn't "handle" it, so whether US or not, the same issue and symptom will be reproducible.

Take the case of a distributed team (with N>1 locales).
You get a slack message "hey buddy, I get this error xxxxx", "ok let me try[time]hmm, I get this error here yyyy" and indeed xxx and yyy are not only different due to the locale so it requires more work for nothing to identify it is the same case.
This is why I think that not detecting broken pipes is fine and helping.

I get what you're saying and also think you have a valid point. Anyways, I think the current message is reasonable and also has a value. The message starts with "broken pipe: ...", which is the most important part and doesn't change between different locales. In the past, we've got many issues opened with people reporting that Jib fails with a broken pipe; obviously many people don't know what it is and thought Jib should fix it.

@chanseokoh fully agree on that point. Maybe we could just stack the error messages of the stack (without the stack traces)? The most complicated part of this task is to reproduce several cases (the one I got is quite random on openshift and I dont own the cluster so cant play much with it :(). So maybe let track this issue and link it when we get connection errors (or related) to refine the message when we get enough "examples"?

Sounds good. Let's leave this issue open for tracking.
Maybe we should try to go simple with just enhancing the current message (which is only for US unfortunately). Whenever there's a failure with an exception, ultimately that is the exception that Jib (and hence Maven and Gradle) throws in the end. Maven and Gradle suggests to check the full stack trace (-X / --stacktrace), which most Java devs do. In the case of broken pipe SocketException, that SocketException will be the lowest one, and any layers on top of it (e.g., SSL) will wrap a lower level exception with an additional context (basically like, "SSL handsake failure because of: broken pipe"). So, for any exception, you can always have the full stack trace (which is not being altered in any way), the whole chain of events, all the code locations where each of these chained exceptions is instantiated, so I feel there's no need to re-invent the wheel.

Closing the issue as part of cleanup -- the existence of this discussion is the documentation in itself. If anyone else encounters a similar issue, we can add an entry to the FAQ.


NUGET.SERVER - server packages from local

I have set up a local server with nuget.server. I can push packages to the repos. When I try to do install like so:
nuget install
Unable to load the service index for source ''
The content at '' is not a valid JSON object
Am i missing something here? The source is defined in Nuget.Config as http://myserver/nuget

Could you paste your entire nuget.config? Feel free to obfuscate secret values but try to leave as much as possible so I can spot any issues.
Also, could you try a nuget.exe command with the package source specified as a command line parameter, -source.


Automatic padframe generation

This is a summary of the discussion at the Hurricane-2 debrief meeting.
This pass would generate a padframe based on user constraints/annotations.  @shunshou has already begun an implementation of this pass: https://github.com/ucb-bar/barstools/tree/master/tapeout/src/main/scala/transforms/pads

Closed due to non-activity. This may belong in a user pass that acts as a flow plugin.


Failed at start script NODE_ENV=dev gulp serve

Greetings,
weather using Nodejs in Visual Studio 2013 pro or trough Node command line,
there is a failure to start. As per the attached npm-debug.log, the problem doe's not seem to stem for npm but most likely with solved-by-flexbox package.
Kinds regards,
Denis

The example software provided does not work with Windows.
It works nicely with Mac or Linux.

Greetings,
your swiftness is much appreciated.
Kind regards,
Denis

Fixed via a8177b96988a2be2ee257252a3a7f606fbf8046f.


Shared drivers readonly: weird behavior

Hi on Windows 10 [10.0.19041.329] I was able to activate enhanced session on Ubuntu 18 but shared drives do not work properly.
I can write file on a shared drive but I cannot edit it anymore:
jj@eva01:~/shared-drives/C:/temp$ uname -a
Linux eva01 5.3.0-59-generic #53~18.04.1-Ubuntu SMP Thu Jun 4 14:58:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
jj@eva01:~/shared-drives/C:/temp$ echo "Test1">test.txt
jj@eva01:~/shared-drives/C:/temp$ echo "Test2">>test.txt
bash: echo: write error: Input/output error
jj@eva01:~/shared-drives/C:/temp$

Printer sharing is disabled.
How can we fix the problem ?

This repo is being archived, so closing this issue.

This repo is being archived, so closing this issue.


add description to all schemas

in v3.1 the number of schemas increased considerably, which can be a bit confusing.

ok, this might be a bit better now.

done.

Thanks for all the effort, @karenetheridge, and @jdesrosiers for the additional reviews!


Leverage Helm Charts in Addons

Where possible, lets replace hard coded YAML files with versioned Helm charts.

Other than ArgoCD, anything else comes to mind? Do you suggest we create helm charts for things that don't have a clear chart, e.g. container insights with fluentd, etc.? It will add maintenance overhead IMO but doable if we get more resources.

I think we try and leverage a chart everywhere possible. Jason shared this https://github.com/aws/eks-charts


please add event after change setting

hi
i need event after update setting fire . i need this event for flash same cache

Unfortunately, that's not possible but PRs are welcomed.


Provide TypeScript definitions

Maybe I am missing something but I have found several references to TypeScript type definitions but none is exposed and/or available in DefinitelyTyped. Is that right? :)

There are some :)
https://github.com/TylorS/reginn/blob/master/packages/reginn/reginn.d.ts

Thanks!


put back the wait mechanism at the horizon

Due to a bug in akka-http the stream has to be closed completely at the horizon (the connection pool got stuck if http request was down during the internal shutdown of the connection).
Their bug is fixed now and the stream can be kept alive even at the horizon.

[ ] also change the wait/remove the wait time for failed TSV retrieve.
[ ] akka-http 10.0.10 fixed another related issue - upgrade the version

How to check: get to the horizon and make sure in the logs that the dc-sync isn't stuck.

This issue is prerequisite to #141. Without it changing the buffered will be very hard (changing the buffers requires changing the state saved at the end to be the last index time and not the token thus requiring another create-consume at the beginning with -1 at the start (to cope the case where there are more infotons at the same index time) and if it will be done on the horizon there will be a lot of redundant null updates)


[Bugfix]Fix bug CTAS with JSON Function

What type of PR is this：

[x] bug
[ ] feature
[ ] enhancement
[ ] refactor
[ ] others

Which issues of this PR fixes ：

Fixes #6498
Problem Summary(Required) ：

This because lateral join json function table slotRef.getTblNameWithoutAnalyzed() is return null.

[FE PR Coverage check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/sql/analyzer/CTASAnalyzer.java
2
2
100.00%


What is the need of this lib

I found people using this for travis but didn't know why and didn't understand the output lcov.info file.
The readme didn't help too much for this too

@okkez Great help, thanks


The Skyline Problem

A city's skyline is the outer contour of the silhouette formed by all the buildings in that city when viewed from a distance. Given the locations and heights of all the buildings, return the skyline formed by these buildings collectively.
The geometric information of each building is given in the array buildings where buildings[i] = [left(i), right(i), height(i)]:
left(i) is the x coordinate of the left edge of the ith building.
right(i) is the x coordinate of the right edge of the ith building.
height(i) is the height of the ith building.
You may assume all buildings are perfect rectangles grounded on an absolutely flat surface at height 0.
The skyline should be represented as a list of "key points" sorted by their x-coordinate in the form [[x1,y1],[x2,y2],...]. Each key point is the left endpoint of some horizontal segment in the skyline except the last point in the list, which always has a y-coordinate 0 and is used to mark the skyline's termination where the rightmost building ends. Any ground between the leftmost and rightmost buildings should be part of the skyline's contour.
Note: There must be no consecutive horizontal lines of equal height in the output skyline. For instance, [...,[2 3],[4 5],[7 5],[11 5],[12 7],...] is not acceptable; the three lines of height 5 should be merged into one in the final output as such: [...,[2 3],[4 5],[12 7],...]
Example 1:

Input: buildings = [[2,9,10],[3,7,15],[5,12,12],[15,20,10],[19,24,8]]
Output: [[2,10],[3,15],[7,12],[12,0],[15,10],[20,8],[24,0]]
Explanation:
Figure A shows the buildings of the input.
Figure B shows the skyline formed by those buildings. The red points in figure B represent the key points in the output list.
Example 2:
Input: buildings = [[0,2,3],[2,5,3]]
Output: [[0,3],[5,0]]
Constraints:
1 <= buildings.length <= 104
0 <= left(i) < right(i) <= 231 - 1
1 <= height(i) <= 231 - 1
buildings is sorted by left(i) in non-decreasing order.

I am Rucha Gosavi I would like to work on this issue

Yes you can! Upload your solution in the "solutions" folder. Your deadline is 22 March'23, 11:59pm


multi: remove RFQ and Taproot Asset Channel functionality for production build of lightning-terminal

We remove any experimental functionality, in order for tapd being compile-time compatible with lnd 0.18.2-beta, for integration into a production build of lightning-terminal.
NOTE: This PR is being merged into the v0-4-0-lnd-18 branch, which will only be used for integration with the upcoming lightning-terminal v0.13.3-alpha.

The diff doesn't seem minimal to me, but I guess that doesn't matter?

Yeah, the diff isn't small. But IMO it's minimal in the number of lines it changes as most of the diff is just cleanly deleting code. So IMO minimal in review effort.

How did you arrive at this diff? I would assume you just needed to set LND's version minimally remove code until it builds, and then disable tap channel support at the interfaces.

That's pretty much what I did, yes :sweat_smile:


revisit http::error::bad_alloc

I dont think catching bad_alloc and turning it into an error_code is a great idea

This error code is no longer used: https://github.com/boostorg/beast/search?l=C%2B%2B&q=bad_alloc

What about adding field_limit like header_limit or body_limit?
Catching bad_alloc in code that calls io_context::run makes it difficult to limit memory for certain basic_parser instance only.

There is no field_limit...


Please help regenerate mocks

Is your feature request related to a problem? Please describe.
Currently, the ExecuteWorkflow mocker can only support return based on the static argument
	mockClient.On("ExecuteWorkflow", mock.Anything, mock.AnythingOfType("internal.StartWorkflowOptions"), mock.Anything,
		&datamodel.NewCustomerMetadata{
			CustomerID:       "good",
		},
	).Return(mockWorkflowRun, nil).Maybe()

	mockClient.On("ExecuteWorkflow", mock.Anything, mock.AnythingOfType("internal.StartWorkflowOptions"), mock.Anything,
		&datamodel.NewCustomerMetadata{
			CustomerID:       "bad",
		},
	).Return(nil, fmt.Errorf("error")).Maybe()

With latest mocker, we could do
	mockClient.On("ExecuteWorkflow", mock.Anything, mock.AnythingOfType("internal.StartWorkflowOptions"), mock.AnythingOfType("string"), mock.Anything).Return(
		func(ctx context.Context, options client.StartWorkflowOptions, workflow string, newCustomerMetadata *datamodel.NewCustomerMetadata) (client.WorkflowRun, error) {
			if newCustomerMetadata.CustomerID == "bad" {
				return nil, fmt.Errorf("error")
			}
			return nil, nil
		},
	).Maybe()

This is becoming very useful if one argument in newCustomerMetadata is timestamp with the usage time.Now(). With existing mocker, it's difficult to do the mocking as testify would find out the argument match by value
Describe the solution you'd like
Please rerun latest mockery
Example
https://github.com/temporalio/sdk-go/pull/1353
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

this will be solved by PRs
https://github.com/temporalio/sdk-go/pull/1353
https://github.com/temporalio/sdk-go/pull/1361


Hide submit button using class instead of left-margin, makes it usable with CSP

Hide submit button using class instead of left-margin, makes it usable with CSP
Description
Replace inline CSS styles with existing bootstrap CSS class to hide the submit button.
I'm still a bit unhappy with the empty paragraph as hack. Without it, the table and the submit button row has no space between them. I can think of two solutions:

Upgrade to bootstrap4 (which has appropriate margin classes) and add them to the table.
Add a static CSS file with a custom class that adds that same margin.

If any of the two seem better to you, please let me know, I'm happy to amend the PR.
Motivation and Context
This makes the default templates usable with a restrictive Content-Security Policy.
How Has This Been Tested?
I could not find any unittests for the UI. If there are, please let me know.
I have manually tested the views, and also tried the enter button of course (that this submit button seems to facilitate).
Screenshots (if appropriate):
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Note that the patch to move to Bootstrap 4 (#379) is only waiting for a final review.

Aha! Excellent. Then I propose we wait wait for that PR and I will adapt mine then.
In the meantime anything else I can do to improve the PR, or is it otherwise fine?

Note that the patch to move to Bootstrap 4 (#379) is only waiting for a final review.

noted

I have updated the branch to use bs4 classes, and updated the changelog as well.
@claudep is there a reason the example project overrides some of the templates from the app? They seem like unnecessary code duplication to me. If you agree, I can make a separate PR for that.

I'm not the author of the example files (and don't care much about them, frankly).

Alright. Then I propose we merge this as is.


BUG: Jax no longer able to see TPUs

Description
Using a TPUVM v3-8 (from TRC). Jax is working fine w/ TPUs but the next day (no actions taken afaik) no longer able to see it.
<EMAIL_ADDRESS>project ID mimetic-slate-353618
TPU name xtpu1
Happy to provide more info
What jax/jaxlib version are you using?
jax 0.3.16, 0.3.15
Which accelerator(s) are you using?
TPUVM v3-8
Additional System Info
No response

Possibly related, though Colab uses older 2VM TPUs: https://github.com/googlecolab/colabtools/issues/3009

Note that what I mean by "no longer able to see TPUs" is that jax.local_devices() only returns the CpuDevice

I'm not saying this is a great solution, but does restarting the TPU VMs help?

I already restarted it (stop and start in tpu console) to no effect. I can check on it again later today

Sorry for the delayed response but they are back when I checked on it today.


Inverse fisheye

Restore graph to pre fisheye state
Use case:
Collapse compound node
Expand it
Perform fisheye
Don't move the node after expand
Collapse again
Reverse fisheye
I.e when focus node position doesn't change between collapse and expand operations

Compound collapse integration ideas (app level):

Have a fisheye mode that locks node positions so they can be expanded and collapsed in the same position -- gauranteeing that inverse fisheye works
when a expanded node is moved, before position move, revert fisheye, after position move end, perform fisheye ( harder to do and more precarious)

Compound expand/collapse on hover

expand complex nodes on hover, collapse them off hover


BLOB STORAGE: Pooled connection observed an error reactor.netty.http.client.HttpClientOperations$PrematureCloseException: Connection prematurely closed BEFORE response

Is your feature request related to a problem? Please describe.
Query/Question
There are no examples for Async api.  I'm currently more interested in BlockBlob stageBlock and flux with replayable.
I'm seeing this in my code.
java.lang.IllegalStateException: The request failed because the size of the contents of the provided Flux did not match the provided data size upon attempting to retry. This is likely caused by the Flux not being replayable. To support retries, all Fluxes must produce the same data for each subscriber. Please ensure this behavior.
Code Snippet:
byte[] byteArray;
ByteBuf buf = Unpooled.wrappedBuffer(byteArray, 0, blockSize);
_blobClient.stageBlock(blockId, Flux.just(buf), blockSize).block();
Describe the solution you'd like
There should be sample examples for async api.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Description Added
[x] Expected solution specified

Hi, @vmaheshw. We will be working to add more samples before we GA. We are aware that they need to be filled out more, and it's just a matter of balancing feature work with samples. But like I said it'll be more complete before GA.
Assuming your blockSize is the same as the size of your byteArray, I'm a bit surprised you're hitting this exception. Are you able to reproduce this consistently? And are you able to share more of a stack trace?

I saw this in different path this time.
2019/08/30 18:22:19.428 ERROR [PooledConnectionProvider]  Pooled connection observed an error reactor.netty.http.client.HttpClientOperations$PrematureCloseException: Connection prematurely closed BEFORE response
2019/08/30 18:22:19.434 WARN [stageBlock]   <-- HTTP FAILED:
2019/08/30 18:22:19.434 INFO [stageBlock] --> PUT
2019/08/30 18:22:23.708 ERROR [PooledConnectionProvider]  Pooled connection observed an error io.netty.util.IllegalReferenceCountException: refCnt: 0
at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1464) ~[netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.buffer.AbstractByteBuf.duplicate(AbstractByteBuf.java:1207) ~[netty-all-4.1.38.Final.jar:4.1.38.Final]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.FluxJust$WeakScalarSubscription.request(FluxJust.java:99) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.netty.channel.ChannelOperationsHandler$PublisherSender.onSubscribe(ChannelOperationsHandler.java:715) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.FluxMapFuseable.subscribe(FluxMapFuseable.java:63) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.FluxMapFuseable.subscribe(FluxMapFuseable.java:63) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.netty.FutureMono$1.subscribe(FutureMono.java:142) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.core.publisher.Flux.subscribe(Flux.java:7799) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:460) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:194) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:749) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:789) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:757) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1031) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:298) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at reactor.netty.FutureMono$DeferredWriteMono.subscribe(FutureMono.java:330) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:3710) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:317) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE]
at reactor.netty.http.client.HttpClientConnect$HttpObserver.onStateChange(HttpClientConnect.java:397) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:501) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:443) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:501) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:443) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at reactor.netty.channel.ChannelOperationsHandler.channelActive(ChannelOperationsHandler.java:112) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:225) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:211) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:204) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelActive(CombinedChannelDuplexHandler.java:414) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:69) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelActive(CombinedChannelDuplexHandler.java:213) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:225) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:211) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:204) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at reactor.netty.tcp.SslProvider$SslReadHandler.userEventTriggered(SslProvider.java:724) [reactor-netty-0.8.3.RELEASE.jar:0.8.3.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:341) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:327) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:319) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1745) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1409) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.handler.proxy.ProxyHandler.channelRead(ProxyHandler.java:255) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:255) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.38.Final.jar:4.1.38.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2019/08/30 18:22:23.711 WARN [stageBlock]  <-- HTTP FAILED:
2019/08/30 18:22:23.714 ERROR [] The request failed because the size of the contents of the provided Flux did not match the provided data size upon attempting to retry. This is likely caused by the Flux not being replayable. To support retries, all Fluxes must produce the same data for each subscriber. Please ensure this behavior.
Basically, the sample code I pasted above is not working with internal retry logic when some network related issue occurs.
So two things to investigate:

Why is the connection closing prematurely?
Why is the retry not working?

@rickle-msft : Should I create another BUG ticket for the crash, to track Async example and the Exception separately.

Sure. It might be better to rename this one for the crash since there's already context here and then open a new on for the examples and just reference this issue since we've already acknowledged that ask

@vmaheshw I've run the code snippet you gave me locally and forced an IO error by unplugging my network connection. I also reran our test for retrying network errors from the commit for this release, and in both cases it behaved as expected.
I have a suspicion that this line is actually underlying most of what's going on:
Pooled connection observed an error io.netty.util.IllegalReferenceCountException: refCnt: 0
IllegalReferenceCountException extends IllegalStateException, which is what we check for in the retry policy to give that extended error message, and we don't retry an IllegalStateException, which explains why you're seeing it fail so quickly. I also think that would explain why the connection gets closed prematurely--we're failing to read from your ByteBuf so we cancel the operation.
The next question, then, is why your ByteBuf has a refCount of 0. Could you try setting a watch point on that value for your ByteBuf object and see where it's getting decremented?

@rickle-msft:
I have a log before stageBlock to show the length of the block size and I already have a non-zero length check.
2019/08/30 18:21:54.198 INFO com.azure.storage.blob.BlockBlobAsyncClient@ Upload block start for blob: 00057 for block size:10524791.
_blobClient.stageBlock(blockIdEncoded, Flux.just(buf), blockSize).block();
After this we call stageblock.block call. One more point, This is not happening every time. It happened after 12 hour run.

@vmaheshw Sorry for not being clear. I wasn't suggesting that the length of the block is zero. ByteBuf's are explicitly reference counted and will throw this exception if somehow someone tries to read from the buffer after the reference count has been decremented to zero because it's effectively been marked for deallocation. I was asking if you'd be able to set a watch point on this reference count value to determine when this value is changing (probably somewhere internal to the sdk).
Something worth noting here is that in preview 3, which will be out in about a week I think, we are changing this api signature to accept ByteBuffer instead of ByteBuf, so hopefully that will fix this issue for free if you'd rather wait for that next preview instead of spending more cycles debugging this now.
@jianghao It's interesting that this is happening only after a 12 hour run. Did you ever encounter anything like this in your stress testing? (TLDR it seems that something is trying to read a ByteBuf with refCount 0 and throwing).

Assigning to @jianghaolu as this appears to be azure core related.
CC: @alzimmermsft

Can this be validated against the latest azure-core that brings in the updated Reactor / Netty dependencies?

I think there is an issue with below code:
byte[] byteArray;
ByteBuf buf = Unpooled.wrappedBuffer(byteArray, 0, blockSize);
_blobClient.stageBlock(blockId, Flux.just(buf), blockSize).block();

My gut feeling is - though the provided Flux.just(buf) is re-playable, the replay emits invalid ByteBuf. The first subscription to the Flux by the underlying http-client (reactor-netty) releases the emitted ByteBuf after writing it to the channel/socket. Such release reduces ref_count by 1. If the code path requires resubscribing to the same Flux then it emits already released ByteBuf i.e. the one with ref_count 0.  Solution is to ensure each subscription to Flux gets valid ByteBuf, i.e. ensure it's really a valid repayable publisher. The just operator alone cannot guarantee this, it has to be combined with Defer:
Flux<ByteBuf> inputFlux = Flux.defer(() -> {
    byte[] byteArray = // init byte[];
    ByteBuf buf = Unpooled.wrappedBuffer(byteArray, 0, blockSize);
    return Flux.just(buf);
});
_blobClient.stageBlock(blockId, inputFlux, blockSize).block();

This way using defer we ensure per subscriber state is valid.

Ok, I can see that the storage blob RequestRetryPolicy is trying to help here a bit so that consumer is not forced to write above defer like logic. The following code in preview2 RequestRetryPolicy:
https://github.com/Azure/azure-sdk-for-java/blob/1b31221c30ea9ede6cfae9a99b79e4b873f01273/sdk/storage/azure-storage-blob/src/main/java/com/azure/storage/common/policy/RequestRetryPolicy.java#L92
Flux<ByteBuf> bufferedBody =
    (context.httpRequest().body() == null)
     ? null
     : context.httpRequest().body().map(ByteBuf::duplicate);

indeed call ByteBuf::duplicate to produce a ByteBuf that shares underlying array. This code seems the root cause of the bug, duplicate will not increase the ref_count, instead it should be ByteBuf::retain which increase the ref_count and it must be followed by ByteBuf::release in doFinally to reduce the ref_count.
I guess, with preview2 above proposed defer will workaround this bug.

Merging the info from my last two comments simplifying them to simple flow. I think below is possible flow causing the error:

User provided a Flux emitting bb1 (this has ref_count 1).
RetryPolicy policy duplicated bb1 to produce bb2. bb2 is simply a reference to underlying array owned by bb1 and has the same ref_count of 1. bb2 is given to httpClient.
HttpClient writes bb2 to wire and reduces ref_count to 0, now both bb1 and bb2 has ref_count 0 (behavior of duplicate).
Write operation in step3 failed hence control go back to RetryPolicy.
RetryPolicy resubscribed to user provided flux that again emits bb1 which ref_count 0.
RetryPolicy duplicated bb1 to produce bb3, both bb3 and bb1 has ref_count 0. bb3 is given to httpClient.
HttpClient sees that bb3 has ref_count 0 and throws IllegalRefCountException.
As Rick mentioned RetryPolicy re throw this as IllegalStateException.
The Channel is closed due to this with PrematureCloseException.

@JonathanGiles I will double check the code to see whether current ByteBuffer based core/storage has similar flaw in different places.

Thanks @anuchandy

@anuchandy Thanks for investigating this. I suspect that since we moved back to accepting ByteBuffers in preview 3, the logic in the retry policy (duplicate) should work now, right?

@rickle-msft right, with ByteBuffer in preview-3, usage of the duplicate operation in RetryPolicy is valid.

@anuchandy @rickle-msft Are there  unit test cases to verify that RetryPolicy is working fine? I'm not able to figure out a way to enforce RetryPolicy other than wait for error from storage account.

I want to be sure that RetryPolicy is working as described. I could not find a way to simulate failure which will automatically enforce the default Retry.
Is there any log/trace message that I can find in my run if Retry was tried. In 10.5V there was a trace message telling Request Number and I could look for "==> OUTGOING REQUEST (Try number='1')". This is not present in V12. So, I'm not confident if this(https://github.com/Azure/azure-sdk-for-java/issues/5180) issue is fixed or not.

@vmaheshw I see. You should be run the tests I pointed you to verify the correct behavior. As for logging the try number, I don't think there is any information on that in the v12 logs right now, but I have included this request in the issue that discusses logging improvements for preview 5 (#4328)

I am going to close this issue as I believe this issue was addressed with the switch to accepting ByteBuffers, and we have not heard anything to the contrary from the customer. Please feel free to reopen or post again if you continue to his this or similar issues.


[GA+Preview] Customer-reported: make GetVectorStoreAsync public again

Issue 240 on openai-dotnet.
This appears to have been unintentionally regressed by the following commit:
https://github.com/joseharriaga/openai-in-typespec/commit/a9b26052016af34d71b40fe2d4a3ccf45e92905e#diff-741e2dd39eecd1c67e6e58d3adb7fcad27c25864e27cc8b786cc364ce6d60ba1R164

Misunderstood; this was one of the intended changes with the introduction of operations.


Using orion-gridFS

Sorry to be so dense, I'm learning meteor and orion by following/modifying a blog example.
I can add orion-gridFS to my meteor application and upload an image of my logo from the admin/dictionary page.
orion.dictionary.addDefinition('logo', 'images',
orion.attribute('file', {
label: 'Site Logo',
optional: true
})
);
How do I access the logo from the layout.html page?
{{ dict 'logo.url' 'http://i.imgur.com/6PeLvIG.png' }} does not seem to work.
Bob

Dear Bob,
Please try: {{ dict 'images.logo.url' 'http://i.imgur.com/6PeLvIG.png' }}
Best,
Elgar


ARROW-12981: [R] Install source package from CRAN alone

I took a stab at implementing the approach @nealrichardson laid out in ARROW-12981. Please let me know what you think, and if you'd like any changes!
I wrote some basic tests for the download_optional_dependencies() helper function, but it would be good to have more comprehensive install tests. These could be something like:
export LIBARROW_BINARY=false
export LIBARROW_BUILD=true
export LIBARROW_DOWNLOAD=false
export LIBARROW_MINIMAL=false

# Make sure offline, feature-light installation works
R -e "install.packages('arrow_x.y.z.p.tar.xz')"
R -e 'stopifnot(arrow::arrow_available(), isFALSE(arrow::arrow_info()$capabilities["parquet"]))'

# Download and install the thirdparty features
R -e "arrow::download_optional_dependencies('arrow-thirdparty')"
source arrow-thirdparty/DEFINE_ENV_VARS.sh
R -e "install.packages('arrow_x.y.z.p.tar.xz')
R -e 'stopifnot(arrow::arrow_available(), isTRUE(arrow::arrow_info()$capabilities["parquet"]))'

@github-actions crossbow submit -g test-r-offline-minimal

Looking through the logs, I'm still downloading XSIMD when TEST_OFFLINE_BUILD is true and ARROW_THIRDPARTY_DEPENDENCY_DIR isn't set. The ARROW_SIMD_LEVEL setting is getting picked up, but somehow that doesn't translate to not using XSIMD.

Looking at https://github.com/apache/arrow/blob/master/cpp/cmake_modules/ThirdpartyToolchain.cmake#L1929-L1935, it looks like you have to set the RUNTIME level to NONE also.

@github-actions crossbow submit test-r-offline-minimal

The build succeeded, though I notice there is still a download step listed:
https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=10731&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=6c939d89-0d1a-51f2-8b30-091a7a82e98c&l=364
I'm digging into this to see if I can figure out what's triggering it

One option for the two-stage testing process would be to do something similar to our version compatibility tests. There we use Github Actions with two jobs, one to write files (here, download the dependencies) and one to read them/test against them (here, install offline). I'm also happy to toss something together for this if you would like + are ok with me sending a couple of commits to your branch.

That would be great!

I also added a few minor comments that I noticed as I read through the PR to see what's been going so far. I will echo @nealrichardson comments that this is a really fantastic PR, in an area of Arrow that is not particularly well documented for contributors, thank you!

Thank you both! I appreciate all the time you've spent providing review and feedback!

@github-actions crossbow submit test-r-offline-minimal test-r-offline-maximal

@github-actions autotune

@github-actions crossbow submit -g r

Oh, I like the idea of creating a fat package and then installing that instead of needing to provide the dependencies separately yourself + env var. Internally, we could use much of the code here to do that as well.
My only (small) concern is making sure it's not easy to confuse which tar is which (though I guess if it's going onto a system without internet it will be pretty clear pretty quickly during install that it's not working).

The conda-osx-clang-py36-r40, conda-osx-clang-py37-r41, and test-r-linux-rchk checks are all known failures and not from this PR.

Yeah, rchk needs a rebase and will be fixed (or when this merges it will be fixed either way)

An idea came to me last night: what if we had a utility that would make a "fat" package, like:
function(source_package) {
  untar(source_package)
  system("inst/download_script.sh tools/thirdparty")
  tar()
}

then you would just copy that arrow_x.y.z.tar.gz and install it, no need to copy other files and set env vars.

I like that! I can take a try at it.

Do you want to include all of the downloaded files (87 MB), or just the ones an R build could possibly use (55 MB)?
Do you still want to be able to run download_optional_dependencies from within an installed R package? If not, we can use the copy of download_dependencies.sh that's in tools/cpp/thirdparty/ (we're currently making another copy for inst/ so it's available at runtime).

A couple of things:

I merged here (because I wanted the fixes from ARROW-13776), but let me know if you want to rebase for a cleaner series of commits.
I removed download_optional_dependencies but didn't update test-r-offline-maximal
Let me know if you want a different name than create_package_with_all_dependencies

I merged here (because I wanted the fixes from ARROW-13776), but let me know if you want to rebase for a cleaner series of commits.

We generally rebase but it's fine here, doesn't look like the diff got messed up. We'll squash-merge in the end so it won't matter.

Tests are currently having errors because of this line:
https://github.com/apache/arrow/blob/6daff455ad1e4c5ac4c84bda5711bdb5c30b6156/r/tools/nixlibs.R#L466
That directory (tools/cpp/thirdparty) would exist if make build had been run. Any suggestions?

Tests are currently having errors because of this line:
https://github.com/apache/arrow/blob/6daff455ad1e4c5ac4c84bda5711bdb5c30b6156/r/tools/nixlibs.R#L466
That directory (tools/cpp/thirdparty) would exist if make build had been run. Any suggestions?

I can investigate later, though that wouldn't explain why the windows builds are failing since that script doesn't get called there

edit: are you sure you want that? the next line checks if (!dir.exists()) and returns

My goal was to figure out if we're building using the downloaded dependencies by checking if the download folder exists in tools/cpp/thirdparty. Whether or not we're building the downloaded dependencies, I thought the tools/cpp/thirdparty folder should always exist, since it gets copied in by make build. This stopifnot was a way to check that assumption. (Put differently, testing if tools/cpp/thirdparty/download is missing doesn't tell me much if tools/cpp/thirdparty is also missing.)
Does that seem reasonable? If you have a cleaner way, let me know!
The windows builds were failing because I hadn't documented all of my arguments. Hopefully they pass now.

I thought the tools/cpp/thirdparty folder should always exist, since it gets copied in by make build.

But tools/cpp is .gitignored, so that check may/will fail in a git checkout, as on CI.
I'll read over the latest iteration and see if I can suggest an alternative.

@github-actions crossbow submit test-r-offline-minimal test-r-offline-maximal

@github-actions crossbow submit test-r-offline-maximal

That build is "passing", but it looks like it didn't actually run any tests.
if(tools::testInstalledPackage("arrow") != 0L) stop("There was a test failure.")
#> Testing examples for package ‘arrow’
#> no parsed files found
#> Running specific tests for package ‘arrow’
#>   Running ‘testthat.R’
#> Warning message:
#> no examples found for package ‘arrow’

@github-actions crossbow submit test-r-offline-maximal

Yeah, looking at the testthat artifact that was uploaded, I realized the workflow was not quite right.

@github-actions crossbow submit test-r-offline-maximal

Aaah, actually the original configuration was fine (though I've adjusted the "Dump test logs" step to always be run (so that it's easier to confirm without downloading the artifacts).
This took a bit of RTFM, but the output of tools::testInstalledPackage() (run against arrow) is placed in arrow-tests of the working directory. So if the tests were not run, the output file would not exist or be blank, but we do see output from it showing that the tests are running. When this run finished I will (double) check that we're not seeing a bunch of skips for various features that are disabled.

Ok, this looks good. Though I'm seeing that S3 was disabled, I'll make sure we've got the dependencies installed on the host so that we can catch that too.
Here are the skips:
══ Skipped tests ═══════════════════════════════════════════════════════════════
• ARROW-11090 (date/datetime arithmetic) (1)
• ARROW-12632: ExecuteScalarExpression cannot Execute non-scalar expression (1)
• ARROW-13364 (1)
• ARROW-13691 - na.rm not yet implemented for VarianceOptions (2)
• ARROW-13799: factor() should error but instead we get a string error message in its place (1)
• Arrow C++ not built with s3 (4)
• Flight server is not running (1)
• Implement more aggressive implicit casting for scalars (ARROW-11402) (1)
• Ingest_POSIXct only implemented for REALSXP (1)
• Minio is not running (1)
• Need halffloat support: https://issues.apache.org/jira/browse/ARROW-3802 (1)
• Need to substitute in user defined function too (1)
• RE2 does not support backreferences in pattern (https://github.com/google/re2/issues/101) (1)
• Sorting by only a single timestamp column fails (ARROW-12087) (1)
• TODO: (if anyone uses RangeEquals) (1)
• Table with 0 cols doesn't know how many rows it should have (2)
• These tests are flaking: https://github.com/duckdb/duckdb/issues/2100 (1)
• This OS either does not support changing languages to fr or it caches translations (1)
• Work around masking of data type functions (ARROW-12322) (1)
• count() is not a generic so we have to get here through summarize() (1)
• empty test (1)
• environment variable ARROW_LARGE_MEMORY_TESTS (2)
• https://issues.apache.org/jira/browse/ARROW-7653 (1)
• packageVersion("stringr") > "1.4.0" is not TRUE (1)
• test now faulty - code no longer gives error & outputs a empty tibble (1)

@github-actions crossbow submit -g r
I'm running the full suite again since we're close and want to make sure this didn't (accidentally) do anything to our other builds

@github-actions crossbow submit -g test-r-offline-minimal

@github-actions crossbow submit test-r-offline-minimal

I'll give this one last read through before merging, but I think this is good to go. Thank you for all this work + taking the journey with us as we found the best way to accomplish this.

Thanks! I learned a bunch doing this.
I had a couple minor questions, following up on comments from @nealrichardson:

Should I swap the argument order for create_package_with_all_dependencies?
Should create_package_with_all_dependencies check ARROW_THIRDPARTY_DEPENDENCY_DIR?

Thank you both for all your help and patience getting this across the finish line!


intel_adsp: hd audio: Add Power Management and Fix Initialization Issues for Intel HDA DAI and DMA Drivers

This pull request includes several improvements to the Intel HDA DAI and HDA DMA drivers. It adds power management support to the HDA DAI driver, ensuring balanced power management calls and correct power domain assignments. The HDA DMA driver is refactored to correct power management initialization and power domain assignments, ensuring proper power state management and reducing power consumption. Additionally, it addresses the invalid initialization sequence and register use in dma_config by ensuring channels are initialized during device initialization. These changes enhance system stability and ensure proper initialization and power management.

@tmleman do we have any tests that are currently failing and that this PR fixes? At least a description of such cases could be helpful

do we have any tests that are currently failing and that this PR fixes? At least a description of such cases could be helpful

@lyakh Failures occur in the internal CI with a wider scope of tests. They can be observed in tests with DMIC and SSP. The reproduction rate is less than 10% (I have never managed to reproduce it locally).
The problem is quite obvious when you look at it closely. The function intel_adsp_hda_channels_init must be called when we raise our device. In the current version, if the device is under a power domain, it will only execute during dma_start, but dma_config is called earlier and unfortunately touches HW registers. To be more precise, the function intel_adsp_hda_set_buffer returns -EBUSY because some registers have incorrect values at the start.
Related commit: 009815e985db5e19b4a91413d2d9da9b265a0c2a

Marking as DNM until I confirm that the change does not introduce regression in SOF.

do we have any tests that are currently failing and that this PR fixes? At least a description of such cases could be helpful

@lyakh Failures occur in the internal CI with a wider scope of tests. They can be observed in tests with DMIC and SSP. The reproduction rate is less than 10% (I have never managed to reproduce it locally). The problem is quite obvious when you look at it closely. The function intel_adsp_hda_channels_init must be called when we raise our device. In the current version, if the device is under a power domain, it will only execute during dma_start, but dma_config is called earlier and unfortunately touches HW registers. To be more precise, the function intel_adsp_hda_set_buffer returns -EBUSY because some registers have incorrect values at the start. Related commit: 009815e985db5e19b4a91413d2d9da9b265a0c2a

@tmleman thanks for the explanation!

Rebase to enable testing in SOF CI: https://github.com/thesofproject/sof/pull/9714

SOF CI did not show any regression. Failures at https://sof-ci.01.org/sofpr/PR9714/build9718/devicetest/index.html look like a platform issue.


npm install / build error "../src/utilities.cc: In function ‘v8::Handlev8::Value format(const v8::Arguments&)’:"

Hello,
I'm trying to install sharp on a CentOS 6.6 machine. What I can see the preinstall script went thru ok.
But installing the sharp fails with following:
><EMAIL_ADDRESS>install /tmp/sharptest/node_modules/sharp
> node-gyp rebuild

make: Entering directory /tmp/sharptest/node_modules/sharp/build
  CXX(target) Release/obj.target/sharp/src/common.o
  CXX(target) Release/obj.target/sharp/src/utilities.o
../src/utilities.cc: In function ‘v8::Handle<v8::Value> format(const v8::Arguments&)’:
../src/utilities.cc:99: error: expected initializer before ‘:’ token
../src/utilities.cc:99: error: expected primary-expression before ‘)’ token
../src/utilities.cc:99: error: expected ‘;’ before ‘)’ token
../src/utilities.cc:103: error: ‘f’ was not declared in this scope
make: *** [Release/obj.target/sharp/src/utilities.o] Error 1
make: Leaving directory `/tmp/sharptest/node_modules/sharp/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:810:12)
gyp ERR! System Linux 2.6.32-504.8.1.el6.x86_64
gyp ERR! command "node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /tmp/sharptest/node_modules/sharp
gyp ERR! node -v v0.10.33
gyp ERR! node-gyp -v v0.10.6
gyp ERR! not ok
npm ERR! weird error 1
npm ERR! not ok code 0

I noticed that my node-gyp was very old and updated that to 1.0.2 just in case, but with no help.

Hi Joonas, if you're using the gcc compiler then v4.6+ is required. I think Centos 6 provides v4.4.
Perhaps try the "dev toolset" http://superuser.com/questions/381160/how-to-install-gcc-4-7-x-4-8-x-on-centos/542091#542091

Excellent! Thanks for the fast reply.
Updating gcc to 4.7 solved this.

Hi guys, I'm receiving the same compiler error on CenOS 6.6 using gcc 4.8.2.
Any other ideas?

@brianviveiros The utilities.cc:99: error: expected initializer before ‘:’ token error is the C++ compiler's way of saying it can't handle the use of a "range-based for loop".
Support for this feature was added in gcc 4.6.
Could something have set the CXX environment variable to override the version you think is in use?

Perfect, that was it!
I followed the install instructions for devtools-2 but didn't export the environment variables.
Thanks!


Upgrade pulumi-terraform-bridge to v3.61.0

This PR was generated via $ upgrade-provider pulumi/pulumi-azuread --kind bridge.

Upgrading pulumi-terraform-bridge from v3.60.1 to v3.61.0.

Ah never mind, I need to do it one more time.


Data Retention

What is the goal / desired outcome?
Be able to purge data, both for data sensitivity and storage cost reasons.
If there is a way to accomplish this today via workaround, what does that require?
Not really, at least it's very awkward.
(Optional) What is your proposal for a feature to solve this?
API
To solve this we'll add a notion of a RetentionSpec which looks like this:
message RetentionSpec {
  // Duration indicates that commits should expire after the duration has
  // elapsed.
  google.protobuf.Duration duration = 1;
  // Transitive indicates that commits should expire when any of their
  // provenant commits does.
  bool transitive = 2;
}

Retention specs will be attached to repos via CreateRepoRequest. Commits that are created on a repo with a retention spec will have their Expires field set based on the details of the spec. Retention specs will also be attached to pipelines, which just passes the retention spec through to the output repo. I don't think we should add a way to set expirations on individual commits, I think it's likely to be confusing and lead to people retaining data they thought would be purged, and I can't think of a compelling use case for it.
Implementation
When a commit expires we unreference the tree associated with it, which eventually leads to its content being GCed. This is similar to what we do in DeleteCommit except that it doesn't recursively delete subvenance, and the commit can still be accessed by *Commit requests, it can't be accessed by *File requests.
To make this work we'll need to add a pfs master, similar to the pps master, that watches the commits collection for expired commits (using an index on the Expires field) and unrefences their trees when the expiration passes. Expiration should block until the commit has been consumed by all of its downstream pipelines.
Usage
Setting a retention spec alone is only going to have the effect the user wants for overwrite workloads where new commits delete old data in addition to adding new data. In an append workload, old commits will expire, but their content will already have been duplicated in later commits. However, setting retention means that when you call delete file and then wait for the expiration time to elapse, the files you deleted will be gone.
There are a couple of other scenarios in which users can inadvertently hold on to data they want to expire. The obvious one is pipelines which copy their input data, users will have to be careful to set such pipelines with a transitive retention spec so that data gets expired when its input does. The other potential issue is if people copy data, there's no particular support for this, users just need to be careful to add retention policies to copied data they don't want to retain.
I've layed out what I think the proto changes for this feature are going to look like in he retention branch: https://github.com/pachyderm/pachyderm/tree/retention

Adding onto this, to make the automated workload smoother we should also make it so that when committing new files, files in the parent commit that are older than the retention policy don't get checkpointed into new commits.

This is a very important feature for large datasets.
I do have a simple point though, it should be possible to manually set or update the retention policy for commits that have finished processing. Maybe even apply them to all the current commits in a repo retroactively?

@jdoliner can you please consider this for 1.11? sounds like one of the more important features you guys thought of.

@RaananHadar we definitely see this as a very important feature of Pachyderm. It's also a very difficult feature, because so much of Pachyderms architecture is built for retaining every version of the data, forgetting data cuts counter to a lot of that architecture. The other thing that makes this tricky is that we're presently working on a rewrite of our storage system which means that if we implement this for the existing system we'll have to reimplement it in the new system which would be duplicating a lot of work. For that reason it makes more sense for us to implement this after the new storage layer lands. We're targeting that for either our next release or the one after, whenever it's ready. And then we'll start working on this immediately. Sorry I can't offer you a better timeline, but that's what makes the most sense for the overall roadmap of the project.

@jdoliner thank you for the explanation. Makes sense.

Looking forward to having this! Not a rush for us though (yet).

Hi,
It would be great if this could just be configured in the pipeline specification. One way would be to specify the TTL in a duration, but for some use cases where the data timing is not yet known in advance it would be useful to be able to specify a number of commits to keep.
Another nice to have would be to still keep track of the provenance such that the deleted data can be reproduced if needed.
Is there already a known workaround? What would happen if we just set a TTL on the objects in the bucket itself - how would pachyderm behave? We're getting closer to this being a blocking issue for us, so any workarounds would be much appreciated.
Thanks for the discussion!

Something that might be relevant to this issue:
Currently we’re dealing with a GDPR issue that seems incoherent to me. The combination of GDPR and standard IRBs for healthcare related studies makes it such that a patient needs to always be able to delete their data. That is, within something like a month after the patient has requested to be removed from a study, all of their data needs to be out of the system. They also require strong data provenance which is obviously competing with the data deletion requirements.
This is probably more of an issue of incoherent regulations, but I thought I’d drop my two cents. Being able to delete data easily ends up being pretty important for dealing with GDPR.

Yeah the law is definitely not clear on these things. I think the overall plan for this feature is that the data will be actually deleted, but the commit/lineage structure will still remain. So you'll have some amount of lineage info still yet the actual data will be gone. There is a lot of complexity here we're still trying to design and sort out so none of what I'm describing is "final" either.
Joey Zwicker
Founder, Pachyderm
On Sat, Jun 13, 2020 at 12:49 PM, jamesthegiantpeach <<EMAIL_ADDRESS>> wrote:

Something that might be relevant to this issue:
Currently we’re dealing with a GDPR issue that seems incoherent to me. The
combination of GDPR and standard IRBs for healthcare related studies makes
it such that a patient needs to always be able to delete their data. That
is, within something like a month after the patient has requested to be
removed from a study, all of their data needs to be out of the system.
They also require strong data provenance which is obviously competing with
the data deletion requirements.
This is probably more of an issue of incoherent regulations, but I thought
I’d drop my two cents. Being able to delete data easily ends up being
pretty important for dealing with GDPR.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub (
https://github.com/pachyderm/pachyderm/issues/4029#issuecomment-643669637
) , or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AANWHQGTUANCQ3UK4VGOTHTRWPJ3JANCNFSM4IQMY2SA
).

Has there been any further discussion/documentation on this? I'm using AWS S3 as the backing store and it would be nice to have a strategy for using intelligent tiering or migration to AWS Glacier.


large bin attack for glibc2.30

The old large bin insertion got two additional checks in glibc2.30.
However, an alternative attack path still exist in latest versions (currently glibc2.30 ~ glibc2.32)
This specific path has existed since before glibc2.23, but I personally have never seen it discussed publicly.

I haven't looked into the technique yet. But by skimming through the technique, I have a question: is this house-of-storm? At least the relevant code snippet is the same as what house-of-storm attacks.
If they are the same, I'd like to rename it to house-of-storm.

TL;DR

Is the relevant code snippet same as house-of-storm?
No. It might seem similar at first glance because both snippets handle similar stuff for different cases.

Is this attack in some sense similar to house-of-storm?
Yes, since house-of-storm is an application of large-bin-attack.

Details
Correct me if I'm wrong, but I believe house of storm is just an application of large bin attack. It might better to just leave it as large bin attack, since there are many possible use case of this primitive aside from house-of-storm.
And just to clarify thing up, the value of this PoC is not a "new exploit vector", but rather to bypass current checks and revive large-bin-attack with an relatively unknown code path.
Generally speaking, there are three major cases when inserting large bin chunks :

victim chunk size < smallest chunk in bin
victim chunk size exist in bin
elsewise

Previous attacks all focus on the 3rd case (this includes house-of-storm and a lot of other challenges). However, starting from glibc2.30, new checks have been enforced and the 3rd case doesn't work anymore. The PoC presented here takes the 1st case instead, which currently has no security checks.

This is confirmed to be working on 2.31. Will be merged after the refactoring.
thanks!

I appreciate that this was added to glibc_2.30 folder. I think it is good to get the the versioning separate to patches to existing attacks.

Should a POC for the House of Storm be added then? This looks like an existing technique, that works very well, that is not in the repo.

@mdulin2 we are doing it. checkout refactor branch :D
Yes. House of Storm should be added. I just don't have the bandwidth to do it.


Rounded web page + One line

When using Rounded Web Page with One Line together the gap below the tab bar is missing if bookmark toolbar is not shown.

Fixed in v1.8.134


Fix 393

Fixes #393
Changes

Rename get_bmi_var_name to get_bmi_output_var_name and make it a protected function
validate name map lookup resolves to an output variable
Removed the -0.0 allowance in C formulation unit test

Testing

Tested with unit tests and local runs using example realizations, but with standalone CFE and CFE+PET multi-bmi

Notes

I had worked up a more generic get_bmi_alias function that validates names/alias by type (output, input, any).  But I'm not sure if it is needed here or not...

Checklist

[x] PR has an informative and human-readable title
[x] Changes are limited to a single goal (no scope creep)
[x] Code can be automatically merged (no conflicts)
[x] Code follows project standards (link if applicable)
[x] Passes all existing automated tests
[x] Any change in functionality is tested
[x] New functions are documented (with a description, list of inputs, and expected output)
[x] Placeholder code is flagged / future todos are captured in comments
[x] Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
[x] Reviewers requested with the Reviewers tool :arrow_right:

Target Environment support

[x] Linux
[x] MacOS

For reference, here is what I was thinking about for a generalized get_bmi_alias function for a Module Formulation
    protected:

        enum BMI_VAR_TYPE{ BMI_INPUT_VAR, BMI_OUTPUT_VAR, BMI_ANY_VAR};

        /**
         * @brief Get correct BMI variable name, which may be the output or something mapped to this output.
         *
         * @param name
         * @param bmi_var_name
         */
        inline void get_bmi_alias(const std::string &name, std::string &bmi_var_name, BMI_VAR_TYPE type)
        {
            std::vector<std::string> names;
            switch(type){
                case BMI_OUTPUT_VAR:
                    //standard output names fi
                    names = get_bmi_model()->GetOutputVarNames();
                case BMI_INPUT_VAR:
                    names = get_bmi_model()->GetInputVarNames();
                case BMI_ANY_VAR:
                    names = get_bmi_model()->GetOutputVarNames();
                    std::vector<std::string> tmp = get_bmi_model()->GetInputVarNames();
                    //dump input names after output names
                    names.insert(names.end(), tmp.begin(), tmp.end());
            }
            if (std::find(names.begin(), names.end(), name) != names.end()) {
                bmi_var_name = name;
            }
            else
            {
                //check mapped/alias names
                std::string mapped_name;
                for (auto & iter : bmi_var_names_map) {
                    if (iter.second == name) {
                        mapped_name = iter.first;
                        break;
                    }
                }
                if (std::find(names.begin(), names.end(), mapped_name) != names.end()){
                    bmi_var_name = mapped_name;
                }
                //else not a BMI recognizable variable for given type
            }
        }

And then get_value would call it, knowing it is looking for valid output vars, like so
get_bmi_alias(output_name, bmi_var_name, BMI_OUTPUT_VAR);


Set up CI with Azure Pipelines

@bethz, for PRs we merge them to staging first then after all the items in a milestone have been pulled into staging we'll do a release to master. See the wiki for a little more detail: https://github.com/Microsoft/Recommenders/wiki/Strategy-to-merge-the-code-to-master-branch
In terms of what you're working on here, do you need to push this first step into staging? Or is it possible to continue to work on it in a separate branch without requiring PR?

I changed the target branch to staging.
May I know more context about this PR? It is not easy to tell directly from the commits. Also, it would be great if a detailed description of the PR can be provided. We have made a template for that and by default it appears when you create a new PR. The description will help reviewers understand the context, content, and other necessary information before performing a quality review. The PR #534 created by @gramhagen may be a good example.

I set up a public pipeline in Azure DevOps and when pointed to Recommenders it automatically created this PR (without saying it did that).  I'll add more detail and look at Scott's references. I appreciate the pointers! thanks.

I'm going to close this PR as I was assigned this as an issue:
https://github.com/Microsoft/Recommenders/issues/425


fetch() streaming

It would be great to know what browsers support fetch streaming because a lot of people are interested in it as a replacement for https://github.com/whatwg/html/issues/2177

I'll close this as a duplicate of https://github.com/Fyrd/caniuse/issues/3173


First pass at a PagerDuty Integration guide

This is a first cut at the docs for this feature.

Need to dark ship this, because reasons.


Issue No. #34 Enhance the user interface.

Enhancing the user interface

Issues Identification
Closes: #34
Description
Summary
Provide a brief description of what this pull request does and why it is needed.
Updation is needed because there were some problem in style.css file.
Details
Include any detailed information about the changes in this pull request.
Change in style.css file.
Types of Changes
Please check the boxes that apply

[x] Bugfix (non-breaking change that fixes an issue)
[ ] New feature (non-breaking change that adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation update (Documentation content changed)
[ ] Other (please describe):

Checklist
Please check the boxes that apply

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] My changes do not break the current system and pass all existing test cases
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

Screenshots

If applicable, please attach screenshots of the changes made to the user interface.
Additional Information
Please provide any other information that is relevant to this pull request.

Disscussed in #131


RFC: prepare for image publishing

@juliocri @chuyd

Codecov Report

Merging #207 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #207   +/-   ##
=======================================
  Coverage   79.17%   79.17%
=======================================
  Files          11       11
  Lines        1508     1508
=======================================
  Hits         1194     1194
  Misses        281      281
  Partials       33       33

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b2aa6d...5551514. Read the comment docs.

@kad done. ENTRYPOINT looks much better indeed.


Fix yq script for setting the security context in the vectordb-minio …

Fixes #69

@pdettori So you apply the security context at the container level and not the Pod right? No other change?
(sorry, I prefer to check if I read it right :smile:)

@guimou correct. One yq command is used to set the security context at the container level. I found that an additional yq command is needed to unset the default security context at the pod level because runAsUser and runAsGroup are set to default values outside OpenShift ranges.

@pdettori Thanks a lot for your PR!


Tests suite crendentials

Hi,
Should we run the project tests (npm test) against our own 3scale application or is there an available application to do so ? I saw that there's one used in Jenkins, can we use it (given you provide the provider_key) ?
Thanks !

The credentials defined in the .travis.yml file are what we use for internal testing, that is why the provider_key is private. But you are free to run the tests using your own 3scale credentials: you just need to set the following environment variables:

TEST_3SCALE_APP_ID
TEST_3SCALE_APP_KEY
TEST_3SCALE_PROVIDER_KEY

The first two should correspond to one of the applications in your 3scale account and the provider key the key that identifies you as a 3scale customer (can be found in the "Account" menu on the top right corner of your admin portal).

Thanks, I was not careful enough when I read the README.md and missed this part : Please note that you will first need to set the following environment variables using your own 3scale keys


there's a tussle between  and Title: but specberus/pubrules think their values must match

Relevant WebAuthn spec source:

Web Authentication:
An API for accessing Public Key Credentials
Level 2

Title: Web Authentication: An API for accessing Public Key Credentials - Level
[ ... ]

I really want the "Level 2" qualification to be rendered below the actual spec title such that wrapping of title makes better sense.
specberus/pubrules think their values must match [1].
But the values of
 and Title: are handled differently, as noted above.
[1] https://www.w3.org/pubrules/?url=http%3A%2F%2Fkingsmountain.com%2Fdoc%2Fwebauthn%2F&profile=WD-Echidna&validation=simple-validation&noRecTrack=false&informativeOnly=false&echidnaReady=true&patentPolicy=pp2004

The handling of Title (and its interaction with Level) aren't determined by Bikeshed at all; all that Title does is set the [TITLE] macro, which is used in boilerplate. Several boilerplates appear to use markup like <title>[TITLE] [LEVEL]</title>, leading to the situation that spec's comment notes. And yup, that's what y'all's personal boilerplate does, so if y'all would rather not have that behavior, y'all can just PR that file, since it'll only affect you.
Regarding the Specberus complaint, I don't think it's the linebreaking that's bothering it, I think it's the use of a - in the Title.

doh, TIL we had "personal" boilerplate...

if y'all would rather not have that behavior, y'all can just PR that file...

So ifn I want to mess around with that file and test it locally, it looks like I can play around with this local file:
~/.local/lib/python3.8/site-packages/bikeshed/boilerplate/webauthn/header.include

thx for your help @tabatkins !

You can also just put a header.include file in the same folder as your .bs file, and Bikeshed will use that instead of one based on your Group.


Add missing flags for the astro check command

Description (required)

Added the flags that are missing from the current documentation for the astro check cli command
Currently, only the --watch flag is documented, however there are several other options available for the command (--minimumSeverity or --preserveWatchOutput for example)

Related issues & labels (optional)

Closes #7542

Hello! Thank you for opening your first PR to Astro’s Docs! 🎉
Here’s what will happen next:

Our GitHub bots will run to check your changes.
If they spot any broken links you will see some error messages on this PR.
Don’t hesitate to ask any questions if you’re not sure what these mean!

In a few minutes, you’ll be able to see a preview of your changes on Vercel 🥳

One or more of our maintainers will take a look and may ask you to make changes.
We try to be responsive, but don’t worry if this takes a few days.

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

Locale
File
Note

en
reference/cli-reference.mdx
Source changed, localizations will be marked as outdated.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.

And @at-the-vr thank you for doing a first review tech check on this! It's so helpful! Take a look at the comments I gave on my editing pass (checking for consistent wording on other parts of the page etc.) and feel free to edit more anytime you want to up your review game and check for things like that! :raised_hands:

Reworded slightly to follow your suggestions and improved the description for that flag


Controller login: use clusterName + url + username to identify the unique logins

No longer needed as name has been removed