Provide streamed access to object data

The BlobReadChannel and BlobWriteChannel implementations buffer data and perform separate RPC calls when they need to flush the buffer. This allows for retries of the transfer operations but uses more billable operations than just writing the data in a continuous stream.

I am not sure I understand the issue.

write-channel only buffer to the minimum block size (256K bytes - see https://cloud.google.com/storage/docs/concepts-techniques#resumable)
read is reading as much as requested or 256K (for minimum transfer per request).

The underlying API for the service (apiary or XML) does not support streaming RPC, so every write/fetch has to be done via a separate RPC call. I expect that to change in the future when the service will support gRPC. Is that what the issue about?

Clarification would be helpful but now I think you were probably referring to the non-resumable write option (e.g. if you were using the apiary client directly you would do something like insertObject.getMediaHttpUploader().setDirectUploadEnabled(true))? And similarly for readers.
If so, then I think we should split the issue to reader and writer.
As for writers it is highly recommended (and also mandated in AE classic) to use resumable writes when content is not small (some examples suggest < 2MB, AE currently limit is 10MB). When the content is small one could write it via the create method (and because the content is small the value of Stream over byte[] is not that meaningful).
As for readers, I think that recommendation is weaker (and AE class constraint is wider 32MB) but still
exists mostly for error handling issues (as you pointed out) and based on my experience when building appengine-gcs-client errors handling and retries made a big difference in the user experience. I guess I could have change "byte[] load"
to return a stream (or to have some way to distinct between a directDownload to resumable/chunk download in the same call though in the case of the former the returned channel/stream would not be serializable) but considering the underlying Apiary client code (https://github.com/google/google-api-java-client/blob/86fd2a574487aaff9b4dbdd0396a98bb2741828f/google-api-client/src/main/java/com/google/api/client/googleapis/media/MediaHttpDownloader.java#L181 & https://github.com/google/google-api-java-client/blob/86fd2a574487aaff9b4dbdd0396a98bb2741828f/google-api-client/src/main/java/com/google/api/client/googleapis/media/MediaHttpDownloader.java#L247) I don't think this is going to be useful as it seems that in the case of directDownload bytes are only transferred as once request was completed.

Something like:
  /**
   * Returns a stream that can be used to read a stored object.
   *
   * @param uri the id of the object to be read
   * @param options options specifying how the object is to be opened; defaults to READ
   * @return a stream for reading the object
   * @throws IOException if a problem occurred accessing the object
   */
  InputStream newInputStream(URI uri, OpenOption... options) throws IOException;

  /**
   * Returns a stream that can be used to write an object.
   *
   * @param uri the id of the object to be written
   * @param options options specifying how the object is to be opened;
   *  defaults to CREATE, TRUNCATE_EXISTING, WRITE
   * @return a stream for writing the object
   * @throws IOException if a problem occurred accessing the object
   */
  OutputStream newOutputStream(URI uri, OpenOption... options) throws IOException;

which provides an idiomatic, stream-based way to upload and download object content. These would map to GET and POST operations that allow the content to be transferred with a single exchange.

See my reply above.
Considering the current state of the Apairy client I see no reason for having these method (though
we could by pass some of the issues by using the XML API instead).
Also, these methods are hiding many GCS specific options or differences (such as passing Object
metadata in write operation, conditional read/write, lack of append mode,...) and therefore I don't think
their place is in this library, however I do think a similar functionality would be nice/useful as part of the
nio.file implementation in a contrib library (as discussed in #46) .

Hmm, I think we should definitely have Input/Output Stream methods. This is how AWS does it, not to mention gcloud-node does this as well.

Channel is a stream. It is part of the newer java nio API. see my comment in #46.

So you're saying that we already have this functionality? What's the gcloud-java equivalent of
AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
S3Object object = s3Client.getObject(
                  new GetObjectRequest(bucketName, key));
InputStream objectData = object.getObjectContent();
// Process the objectData stream.
objectData.close();

I think the main discussion here is about a request for a stream based one RPC call (non-serialize/non-resumeable) vs current state which provides a serializable and resumeable stream based on multiple resumable RPCs. I agree that if that is indeed the topic discussed here the title is wrong as we do provide a "streamed access to object data".

see DownloadAction in the storage example: https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/src/main/java/com/google/gcloud/examples/StorageExample.java
        try (BlobReadChannel reader = storage.reader(blob.bucket(), blob.name())) {
          WritableByteChannel channel = Channels.newChannel(writeTo);
          ByteBuffer bytes = ByteBuffer.allocate(64 * 1024);
          while (reader.read(bytes) > 0) {
            bytes.flip();
            channel.write(bytes); // or do whatever with the bytes read.
            bytes.clear();
          }
        }

And if for any reason one prefers not to use java.nio and use InputStream instead he can do:
InputStream input = Channels.newInputStream(storage.reader(blob.bucket(), blob.name()));

Channels are not as well supported as streams and there is no downside to providing both.
    Path localData = Paths.get("localData.txt");
    try (OutputStream os = storage.newOutputStream(TEST_OBJECT)) {
      Files.copy(localData, os);
    }

That's setting aside the implementation difference where the Blob channel performs multiple (billed) operations against GCS rather than just one.
Also BlobReadChannel is wrapping HttpResponse objects that can already provide a InputStream over their content directly.

I would replace "supported" with "adopted" and even that I might not fully agree with.
Given that it is very easy to get an InputStream from a Channel (using a standard API)
I don't see a reason to provide both. However, I don't mind to provide an InputStream instead of a Channel (that is a minor change with a minor [nagtive in my mind] value, though the negative part
is slightly higher with OutputStream vs writer Channel as Buffers that go with Channels are more widely accepted).

So, it seems that there are various issues discussed here. Please decide what this topic this issue was create for and create other issues for the other (related) topics.

As mentioned above (multiple times), the apiary client does not expose the HttpRespose stream directly.

One more data point to the nioChannel vs io.Stream discussions (which is minor in my mind as they are both "streams" can be trivially converted from one another) is that it looks like grpc-java is based on Netty which is based on nio. I am not sure how much of the nio would be visible from the grpc layer but if something does it is more likely to efficiently work with other nio related classes.

@jgeewax I hope this addressed your concern. We do provide a streaming API to read a content of a Blob and we are using the newer Java standard for it (I assume Amazon API preceded nio but that
should not be a factor).  Converting between the two APIs is trivial and provided by the Java standard APIs. I consider this issue to be nio.Channel vs io.Stream and I plan to close as "wontfix".
The more I think about it the more I am convinced that providing the Channel would be preferred.
Here are some of my reasoning:

The nio is the new Java IO API and there is no good reason to stick with the older one (newer framework, libraries are more likely to support the latter - also see (2)).
Channels are provided with both Synchronous and [Asynchronous] (http://docs.oracle.com/javase/7/docs/api/java/nio/channels/AsynchronousChannel.html) flavors for when we provide an asynchronous API calls.
App Engine GCS client API which has significant number of users (and traffic) is also using Channels, so migration will be smother.

OK if this is really just a difference between nio.Channel and io.Stream it sounds like it's "easy enough" for people to convert between the two. Some more questions...

Do we have any idea of how annoying this is to external people?
Are we shoving them in a direction that we need not necessarily shove them?
Would we incur additional maintenance cost by virtue of having these methods tossed in for convenience?

@jboynes , any opinions on these? @aozarov ?

I don't think this will "annoy" people. nio was part of Java 7 (which this library specify as minimum requirement) and Java 7 is out there since [2011] (http://en.wikipedia.org/wiki/Java_version_history#Java_SE_7_.28July_28.2C_2011.29). That was never a complaint by people that were using the appengine_gcs_client.
I think using the channel is the right directions for the reasons I mentioned above (and the conversion is a trivial one-liner, and for that reason I don't think we should pollute the API with 2 flavors that provide the same, and only channels would be able to support the async option in the future).
I am closing this issue (feel free to re-open if you think it should be further discussed) and I am going to create a separate issue for using one RPC for read-write vs using chunks (multiple resumable RPC calls).


refactor: rename package with rollup-plugin prefix

does this affect how you reference it or just for consistency w/ something in the package name?

this is for consistency https://rollupjs.org/guide/en#conventions


Create Better Documentation for Kubectl and Tooling

Proof of concept for new kubectl documentation:
https://pwittrock-kubectl.firebaseapp.com/pages/generators.html

@soltysh @seans3 @kubernetes/sig-cli-maintainers

/remove-lifecycle stale

@pwittrock @seans3 given the new format of the kubectl book is this still valid? I'm aware we still need to update the docs, but I don't think we need general issue. If anything I'd prefer us having specific areas called out that need work, wdyt?

/remove-lifecycle stale


[Fix-7892][api] Fixed when a non-admin user creates a udf function, resources other than .jar will be displayed in the UDF resource

Fixed when a non-admin user creates a udf function, resources other than .jar will be displayed in the UDF resource
This closes #7892

Verify this pull request

This pull request is already covered by existing tests, such as (please describe tests).
org.apache.dolphinscheduler.api.service.ResourcesServiceTest#testQueryResourceList

Codecov Report

Merging #8458 (5a8c9e4) into dev (df6eef8) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                dev    #8458      +/-   ##
============================================
+ Coverage     45.34%   45.35%   +0.01%
- Complexity     4025     4029       +4
============================================
  Files           686      686
  Lines         26702    26704       +2
  Branches       2867     2867
============================================
+ Hits          12107    12112       +5
+ Misses        13450    13449       -1
+ Partials       1145     1143       -2

Impacted Files
Coverage Δ

...heduler/api/service/impl/ResourcesServiceImpl.java
54.34% <100.00%> (+0.46%)
:arrow_up:

...dolphinscheduler/remote/future/ResponseFuture.java
83.60% <0.00%> (+1.63%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df6eef8...5a8c9e4. Read the comment docs.


Fix typo and grammatical error on line 164

"Past" should be "Paste" & "back" replaced with "link"

I guess you like typos?

I am sorry. I am a bit rusty on my GitHub skills. I thought I committed your changes to the master.
Apologies for any misspellings and all abrupt emails as this was sent from my phone.

From: Derek J. Fields Jr<EMAIL_ADDRESS>Sent: Wednesday, April 24, 2019 10:35:39 AM
To: MMEnter/CompactView
Cc: MMEnter; State change
Subject: Re: [MMEnter/CompactView] Fix typo and grammatical error on line 164 (#18)
I guess you like typos?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHubhttps://github.com/MMEnter/CompactView/pull/18#issuecomment-486294328, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABGU52GTACD6Q72HL4G4OLDPSB44XANCNFSM4GKTPGCA.


How to know checkbox selected or unselected event?

  var tree = $("#tree").tree({
    primaryKey: "id",
    uiLibrary: "bootstrap4",
    dataSource: [{ id: 1, text: "aa" }],
    checkboxes: true,
    select: function (e, node, id) {
      console.log(id);
    },
  });

  tree.on("select", function (e, node, id) {
    console.log(id);
  });

Use the checkboxChange event. You can see example at https://gijgo.com/tree/events/checkboxChange


Update JpaPersistService to ignore subsequent calls to start() and stop()

This resolves #947 and supersedes #972 and #598. It possibly also supersedes #1132, although I'm not sure what that user's use case is.
The current behaviour is to throw an IllegalStageException when JpaPersistService.start() or JpaPersistService.stop() are called multiple times. However, this clashes with the javadoc in PersistService and UnitOfWork, both of which state that subsequent calls to these methods are no-ops.
JpaPersistService.end() already handles multiple calls, and has the following comment:
// Let's not penalize users for calling end() multiple times.
This PR brings JpaPersistService and related tests into line with the documentation and resolves a pain point that multiple people (including myself) have experienced, e.g. https://stackoverflow.com/questions/17402081/how-to-start-jpa-in-a-guice-quartz-web-application.
Thank you for taking the time to look at my PR.

Thanks Chris ; I came here to fix the same problem... And saw you already did the job. I hope we'll get a release soon :)

@jlannoy You're more than welcome, I hope we do too :) The PR got approved along with a few others recently, so we'll see what happens!


[Feature Request] PDF-to-image to support custom page selection

It would be great if we could send a PDF to the image API and just get a certain single page image back. That would allow us to use the service as a thumbnailer

We should add an option only to convert certain pages to images.

Can I work on this task.

Sure!


Push resources

Depends on #7
User Flow:

Reactor Pulls down resources ⬇
Human makes changes 👷♂
Reactor Pushes changes ⬆

That human element in the middle is a way to do resource changes back to launch.adobe.com without having to use merge-conflict features.

@mikeumus would be great to see an updated README as well in here for both PRs

@DavidLozzi, good call, will do! 📝👍


Configurable timeout and keep-alive interval

Just a small PR I whipped up to allow the configuration of the timeout and keep-alive interval values. Fixes #1 .

Thanks for doing this! I edited the documentation and sped up the tests somewhat. Released as v0.7.1.


The action 'Select All' or 'Select All in Page' doesn't take effect for one newly created table

Storage Explorer Version: 1.29.0-dev
Build Number: 20230414.2
Branch: main
Platform/OS: Windows 10/Linux Ubuntu 22.04/MacOS Ventura 13.3.1 (Apple M1 Pro)
Architecture: ia32/x64
How Found: From running test case
Regression From: Previous release (1.28.1)
Steps to Reproduce

Expand one storage account -> Tables.
Create a table -> Add one entity.
Click 'Select All'.
Check whether the entity is selected.

Expected Experience
The entity is selected.

Actual Experience
The entity is not selected.

Additional Context
This issue doesn't reproduce if reopen the table.

Verified this issue on the build 20230416.3. Reproduce. Will verify it on a newer build.

Verified this issue on the new build 20230418.4. Fixed.


containers: deployments query

a graphql query deployments should be available. it should return the list of deployments installed on a device
input: deviceId
output: [deployment]

implemented in #678


GH-PAGES

could you please help me to understand the purpose of gh-pages.yml file ? it seems like we can host this project directoly on github pages weather its streamlit web ui or the api inferance ?
i was actually trying to learn about github pages and actions. could you plz tell a little bit abou this? it will help me to understand your projeect in more better way and thanks alot!

This file is only used to generate help documentation, not for streamlit or API purposes.

anyone has created any whatsapp telegram or discord bot with your api ?

"https://huggingface.co/spaces/HarryLiu/MoneyPrinterTurbo"
thank you can you tell how can i secure my api keys while keeping my space as public? even if i duplicate this space and input my own api keys in config.toml it will give error!


Feature Request: Add "details" tag to whitelist

I'm a little surprised that the html5 "summary" tag is whitelisted, but the "details" tag (that it is used with) is not whitelisted:
http://html5doctor.com/the-details-and-summary-elements/
Might it be possible to include the "details" tag in the white list?  I think this could be a really useful feature

Closed by #171


Render @error response as 4XX in OpenAPI 3

This PR changes the handling of @error by the OpenAPI 3 emitter to produce a "4XX" response rather than "default".
This is not removing any capability from the user -- users that want "default" can use the @defaultResponse decorator.
Fixes #1344

Closing as design is still evolving.


Removing entries from "Recently opened" picker closes the picker

VSCode Version: 1.27 insiders
OS Version: macOS

Steps to Reproduce:

open the recently opened quick pick
remove files by clicking on the "X" multiple times

=> 🐛 at random times the picker decides to close

When you click quickly the first click triggers the toolbar item and the second triggers a double click event in the list widget which then triggers a selection change that makes the picker close with the selected item as the result.
@joaomoreno Any ideas? Should we add an option to the list widget to disable selection on double click?

When you click quickly the first click triggers the toolbar item and the second triggers a double click event in the list widget

It sounds like the event shouldn't even reach the list, if it is supposed to be handled by the toolbar. The toolbar action should stopPropagation.

ActionBar already stops propagation: https://github.com/Microsoft/vscode/blob/43676d0cb7db94e2fa771f1d10bff10eaa01b453/src/vs/base/browser/ui/actionbar/actionbar.ts#L126
That does not seem to affect the double click event, which is listened on here: https://github.com/Microsoft/vscode/blob/43676d0cb7db94e2fa771f1d10bff10eaa01b453/src/vs/base/browser/ui/list/listWidget.ts#L455

The event which needed to stop propagation was dblclick.


Ignore devices in the log/add support for Wyze Doorbell Pro Cam

Is your feature request related to a problem? Please describe:
Yes, the Wyze Doorbell Pro Cam is not supported, but the plugin sees the camera at every refresh interval. I put in the mac address of the chime and the camera to be ignored in the plugin settings, but it still floods the log every 3 minutes that there is an unsupported device.
Or can support be added for the camera?
Describe the solution you'd like:
A way to disable unsupported devices from being placed in the homebridge log
Describe alternatives you've considered:
Additional context:
[8/5/2023, 3:05:38 PM] [Wyze] Camera (Model: GW_BE1 (Mac: GW_BE1_7C78B2A33FE8 (Name: Front Door) not supported submit a request to https://github.com/jfarmer08/homebridge-wyze-smart-home/issues to have it added. If no issues are found
[8/5/2023, 3:08:39 PM] [Wyze] Camera (Model: GW_BE1 (Mac: GW_BE1_7C78B2A33FE8 (Name: Front Door) not supported submit a request to https://github.com/jfarmer08/homebridge-wyze-smart-home/issues to have it added. If no issues are found
[8/5/2023, 3:11:39 PM] [Wyze] Camera (Model: GW_BE1 (Mac: GW_BE1_7C78B2A33FE8 (Name: Front Door) not supported submit a request to https://github.com/jfarmer08/homebridge-wyze-smart-home/issues to have it added. If no issues are found
[8/5/2023, 3:14:40 PM] [Wyze] Camera (Model: GW_BE1 (Mac: GW_BE1_7C78B2A33FE8 (Name: Front Door) not supported submit a request to https://github.com/jfarmer08/homebridge-wyze-smart-home/issues to have it added. If no issues are found
[8/5/2023, 3:17:40 PM] [Wyze] Camera (Model: GW_BE1 (Mac: GW_BE1_7C78B2A33FE8 (Name: Front Door) not supported submit a request to https://github.com/jfarmer08/homebridge-wyze-smart-home/issues to have it added. If no issues are found
[8/5/2023, 3:20:41 PM] [Wyze] Camera (Model: GW_BE1 (Mac: GW_BE1_7C78B2A33FE8 (Name: Front Door) not supported submit a request to https://github.com/jfarmer08/homebridge-wyze-smart-home/issues to have it added. If no issues are found

I believe there is a logging option in the plugin configuration that you can set to "none" which will at least get rid of the flood of messages. The filter should be ignoring the device completely so that is unexpected. Will you please post your plugin config from the homebridge json? As far as adding support, what kind of support is of interest? This plugin does not support video streams of any kind and other projects are more appropriate for that work. If non video features are in question definitely note them here to be considered.

Just looked into this. And it's not following the standard logging function. It will print no matter what. I will try to address this soon, but it does not cause any issues.

I believe there is a logging option in the plugin configuration that you can set to "none" which will at least get rid of the flood of messages

It currently is set to none, but it is not following the rules currently. As also noted jfarmer08 said he would look into this.

As far as adding support, what kind of support is of interest?

I know this plugin isn't for video streams, but if a toggle to turn it on or off would be possible, that would be great. There are toggles with the regular Wyze cams I have, but not the Doorbell Cam.

@jfarmer08 are you okay to change this logging to be under the debug option? If so I can go through and change it in a PR. I do like how it makes it obvious when a product needs support and not hiding that in debug. Maybe an additional option of some kind?
@ExioStar that makes sense. The thing is, the video doorbell pro doesn't have an 'on/off' functionality like the other cameras since it is battery powered (even if hardwired). There are some efforts looking into the VDP in other projects like docker-wyze-bridge

@carTloyal123  I am working on a few changes, also adding more support for Camera Flood light and Siren.

https://github.com/jfarmer08/homebridge-wyze-smart-home/releases/tag/v0.5.37-alpha.6


iOS WebThreadLock crash on barcodeScanFailed

I recently upgraded to 6.0.1 from a much older version (~2013), and have begun experiencing an occasional crash on iPhone, especially when scanning in landscape mode.
I think I may have figured it out, but I'm pretty new to iOS and mobile programming, and am only reading about dispatch queues this morning! (I did not write the app I'm working on, just modifying it.)
The debugger message is

bool _WebTryThreadLock(bool), 0x15531220: Tried to obtain the web lock from a thread other than the main thread or the web thread. This may be a result of calling to UIKit from a secondary thread. Crashing now...

The top section of the call stack is
1   0x2836cee7 WebThreadLock
2   0x2968ea11 <redacted>
3   0x92811 -[CDVViewController shouldAutorotateToInterfaceOrientation:]
4   0xa0dd5 -[MainViewController shouldAutorotateToInterfaceOrientation:]
5   0x92903 -[CDVViewController supportedInterfaceOrientations]
6   0x2929ca67 <redacted>
7   0x2929c581 <redacted>
8   0x29321d9d <redacted>
9   0x29abed37 <redacted>
10  0x29574bdd <redacted>
11  0x2959a6d5 <redacted>
12  0x29358aef <redacted>
13  0x292fcd19 <redacted>
14  0x29599a89 <redacted>
15  0x292fc8bd <redacted>
16  0xa2a3f -[CDVbcsProcessor barcodeScanDone:]
17  0xa30bf -[CDVbcsProcessor barcodeScanFailed:]

The actual line that's crashing is in source file CDFVViewController.m, on the line where res is set to the result of the webView.stringByEvaluatingJavaScriptFromString method (last line of this snippet):
- (BOOL)shouldAutorotateToInterfaceOrientation:(UIInterfaceOrientation)interfaceOrientation
{
    // First, ask the webview via JS if it supports the new orientation
    NSString* jsCall = [NSString stringWithFormat:
        @"window.shouldRotateToOrientation && window.shouldRotateToOrientation(%ld);"
        , (long)[self mapIosOrientationToJsOrientation:interfaceOrientation]];
    NSString* res = [webView stringByEvaluatingJavaScriptFromString:jsCall];

To help debug, I put in logging statements at the begin and end of all of the orientation-related methods, as well as "captureOutput", and noticed this interesting thing:
2016-08-23 13:10:44.953 Sappi Stock Manager[2412:1225939] begin CDVViewController.shouldAutorotateToInterfaceOrientation
2016-08-23 13:10:44.953 Sappi Stock Manager[2412:1225939] about to return from CDVViewController.shouldAutorotateToInterfaceOrientation
2016-08-23 13:10:44.954 Sappi Stock Manager[2412:1225939] begin CDVViewController.shouldAutorotateToInterfaceOrientation
2016-08-23 13:10:44.954 Sappi Stock Manager[2412:1225939] about to return from CDVViewController.shouldAutorotateToInterfaceOrientation
2016-08-23 13:10:45.450 Sappi Stock Manager[2412:1225939] Orientation change event.
2016-08-23 13:10:45.450 Sappi Stock Manager[2412:1225939] Footer positioned at: 528
2016-08-23 13:10:45.451 Sappi Stock Manager[2412:1225939] Subtracting menu height of: 50
2016-08-23 13:10:45.451 Sappi Stock Manager[2412:1225939] Calculated final height of: 518
2016-08-23 13:10:46.851 Sappi Stock Manager[2412:1225972] Inside captureOutput
2016-08-23 13:10:47.263 Sappi Stock Manager[2412:1225972] begin CDVViewController.shouldAutorotateToInterfaceOrientation
2016-08-23 13:10:47.264 Sappi Stock Manager[2412:1225972] bool _WebTryThreadLock(bool), 0x15531220: Tried to obtain the web lock from a thread other than the main thread or the web thread. This may be a result of calling to UIKit from a secondary thread. Crashing now...

All prior calls to CDVViewController.shouldAutorotateToInterfaceOrientation take place on thread 1225939, with the statement "Inside captureOutput" taking place on various other threads. But notice that the call to CDVViewController.shouldAutorotateToInterfaceOrientation which fails occurs on thread 1225972, the same thread as the last "Inside captureOutput" statement!
I'm assuming 1225939 is the main thread, and captureOutput gets called in a secondary thread. I see in CDVbcsProcessor.setupCaptureSession:
[output setSampleBufferDelegate:self queue:dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_LOW, 0)];

Then I see in CDVbcsProcessor.barcodeScanSucceeded:
- (void)barcodeScanSucceeded:(NSString*)text format:(NSString*)format {
    dispatch_sync(dispatch_get_main_queue(), ^{
        [self barcodeScanDone:^{
            [self.plugin returnSuccess:text format:format cancelled:FALSE flipped:FALSE callback:self.callback];
        }];
        AudioServicesPlaySystemSound(_soundFileObject);
    });
}

But CDVbcsProcessor.barcodeScanFailed does not dispatch back to the main thread:
- (void)barcodeScanFailed:(NSString*)message {
    [self barcodeScanDone:^{
        [self.plugin returnError:message callback:self.callback];
    }];
}

I feel like this is the problem, and I'm going to test that now. I'll update this note after I see what that does.

I am happy to report that I appear to have fixed the problem!
I simply replaced this code in CDVBarcodeScanner.mm
- (void)barcodeScanFailed:(NSString*)message {
    [self barcodeScanDone:^{
        [self.plugin returnError:message callback:self.callback];
    }];

with
- (void)barcodeScanFailed:(NSString*)message {
	dispatch_sync(dispatch_get_main_queue(), ^{
    	[self barcodeScanDone:^{
        	[self.plugin returnError:message callback:self.callback];
    	}];
    });
}

I confirmed that when barcodeScanFailed is called now (which it still is sometimes, but that's a different issue), it is still in the same secondary thread as the capture session, but its call of barcodeScanDone now takes place in the main thread, as do all succeeding calls to the orientation-related view methods. So now the scan failure is just handled normally, not by crashing.

@JohnBrooking That's great news!

Sure, if you agree that it's correct.

I do, it makes a lot of sense to me. That and the fact you no longer see the crash 😉

@JohnBrooking  @EddyVerbruggen I didn't see the JohnBrooking's PR nor the closed PR's

@JohnBrooking thanks John, I see you never sent a PR so I added it into the plugin for you.

Thanks, guys. Sorry I dropped it. I'm new to Github/open source, and I didn't know "PR" meant I was supposed to do something. :blush:

No problem @JohnBrooking. The nice thing about open source is if you find a bug you can always send a pull request.


When destination SIT delivery date is changed the actual delivery dat…

…e also changes
Description
When a destination SIT is delivered and the date out is changed, the actual delivery date of the shipment will change as well.
Setup
Make a DESTINATION SIT request. Get to the point where your destination SIT is in DELIVERED state with a date out. The actual delivery date should match the date out. Edit the date out and save the SIT request, and the actual delivery date will change as well.
References

Pivotal story for this change

I'm using the shipment ENTDEL to deliver the shipment and SIT. When I edit the out_date, I get an "unprocessable entity". Not sure if that is happening due to data issue, this PR, or a previous issue.

Just tested same shipment on my machine and worked. Maybe repopulate your database with fresh data. It still doesn't work, let me know and we can look at it.

Can you add an integration test to check this functionality?

yes, that's a good idea. will do

I got my db back working and the actual delivery date is successfully changing when the SIT out date is edited.  However, If I go to change the actual delivery date, it shows up as the outdate.  It appears that it is actually getting edited, but it switches in the UI to whatever the SIT outdate is.
A side thing I noticed, since I know you fixed this elsewhere-- When filling out the "Release from SIT" form, the text input is bold.  If it's super quick fix on your end to fix this, it would be great if you fixed it here.  Otherwise, I can create a story for it.


Reason for sanctum X-XSRF-TOKEN mismatch code 419

Laravel Version
^9.19
PHP Version
8.1
Database Driver & Version
mariadb 10.3
Description
Because browser urlencoded cookie value, but middleware VerifyCsrfToken not do proccess with urldecode, so not match.
To resolve this bug.
Open File Illuminate\Foundation\Http\Middleware\VerifyCsrfToken
And we can replace this function in middleware VerifyCsrfToken :
protected function getTokenFromRequest($request)
{
$token = $request->input('_token') ?: $request->header('X-CSRF-TOKEN');
if (! $token && $header = $request->header('X-XSRF-TOKEN')) {
try {
$token = CookieValuePrefix::remove($this->encrypter->decrypt($header, static::serialized()));
} catch (DecryptException $e) {
$token = '';
}
}
return $token;
}
With this new function :
protected function getTokenFromRequest($request)
{
$token = $request->input('_token') ?: $request->header('X-CSRF-TOKEN');
if (! $token && $header = $request->header('X-XSRF-TOKEN')) {
try {
$token = CookieValuePrefix::remove($this->encrypter->decrypt(urldecode($header), static::serialized()));
} catch (DecryptException $e) {
$token = '';
}
}
return $token;
}
Steps To Reproduce
See https://github.com/laravel/framework/issues/46650#issuecomment-1571279254

Hey there, thanks for reporting this issue.
We'll need more info and/or code to debug this further. Can you please create a repository with the command below, commit the code that reproduces the issue as one separate commit on the main/master branch and share the repository here?
Please make sure that you have the latest version of the Laravel installer in order to run this command. Please also make sure you have both Git & the GitHub CLI tool properly set up.
laravel new bug-report --github="--public"

Do not amend and create a separate commit with your custom changes. After you've posted the repository, we'll try to reproduce the issue.
Thanks!

Hey there,
Unfortunately we don't support this version anymore. Please check out our support policy on which versions we are currently supporting. Can you please try to upgrade to the latest version and see if your problem persists? If so, please open up a new issue and we'll help you out.
Thanks!


Fix optional source path declared in included file with non-relative paths.

Given the following configuration, the path MyApp/Tests/Shared will not be added to the target even if the path exists.
Shared/templates.yml
targetTemplates:
    Target:
        sources:
            - path: Tests/Shared
              optional: true

MyApp/project.yml
include:
    - path: ../templates.yml
      relativePaths: false

targets:
    MyApp:
        templates:
            - Target

HI @ileitch. Would you be able to add a unit test that validates the fix


[5.8] Fix Arr::only exception when passing invalid keys

Fix Arr::only exception when passing Invalid Keys , for example when passing something like this will throw this exception
$invalidKeys = [1.5, null, ['key' => 'value'], new stdClass];

Arr::only($array,$invalidKeys);

I feel like this adds unnecessary overhead to the function.


Add e2e tests for Windows

Changes

Two new tests have been added for Windows TaskRuns for testing
success and failure states.
/kind misc
Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[ ] Docs included if any changes are user facing
[x] Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes

NONE

/assign @dibyom

/ok-to-test

Thanks for the feedback, those changes have been added 🙂

/test pull-tekton-pipeline-alpha-integration-tests

/lgtm


Adding wkhtmltopdf here.

@smn @KaitCrawford ready for review.

LGTM but I think @smn should have the final say


Send arbitrary amount of $ada in multi asset minting tx

Hi!
I'm a big fan of this library, that I've already used in two project live on mainnet.
I've been minting some NFT with the intructions at: https://github.com/bloxbean/cardano-client-lib#token-minting-transaction.
In particular I want to mint the token directly into my recipents wallet.
Everything is working fine, but I do have an additional requirement, can I send more $ada in that tx?
Let's say I mint 1 token and instead of sending just 1.4 $ada (or thereabout) I want to directly send, let's say 10 $ada?
Is there a way I can do that? (is it even possible at blockchain level?)
Thanks 🙏

Thanks.

Let's say I mint 1 token and instead of sending just 1.4 $ada (or thereabout) I want to directly send, let's say 10 $ada?

This is an interesting usecase. Unfortunately,  the current high-level api (MintTransaction) doesn't have a "ada" value field.
Also, there is no high level api available to send both MintTransaction and PaymentTransaction in one transaction request.
Probably, a clean solution for future release :- Add a "value" field in MintTransaction to send any extra Ada in the mint transaction.
For now you have two options:
Option 1:
Use low level  apis like TransactionInput / TransactionOutput. But in that case, you have to select utxos and build the Transaction in your application.
Option 2
Set min uxto value in TransactionDetailsParam to your required lovelace amount. Currently, default value is 1 Ada.
Pass the TransactionDetailsParam as part of mintToken().
Note: Currently, some extra ada value is getting added if you send a custom minAda. May be some unwanted side-effect due to change in min ada value in the client library which I need to debug.
But you can adjust the lovelace amount in TransactionDetailsParam based on your testing to avoid any extra ada value in the output.
Hope that helps.
Just wondering, if your project is public, would you be interested to add your project in the "Used By" section of this project.
MintTransaction paymentTransaction =
                MintTransaction.builder()
                        .sender(sender)
                        .receiver(receiver)
                        .mintAssets(Arrays.asList(multiAsset))
                        .policyScript(scriptPubkey)
                        .policyKeys(Arrays.asList(skey))
                        .build();

        TransactionDetailsParams detailsParams = TransactionDetailsParams.builder().ttl(getTtl()).build();
        detailsParams.setMinUtxoValue(new BigInteger("3000000"));

        BigInteger fee = feeCalculationService.calculateFee(paymentTransaction, detailsParams, metadata);
        paymentTransaction.setFee(fee);

        Result<TransactionResult> result = transactionHelperService.mintToken(paymentTransaction, detailsParams, metadata);

Thanks for the quick answer.
I'll play around in testnet with the recommended workarounds.
Of course I can add the projects using the lib.
If that's ok, I'll keep this issue open and I'll report findings.
Are you planning to implemnt in java the cddl part? Or you'll be happy to rely on the native library?
Thanks again!

If that's ok, I'll keep this issue open and I'll report findings.

Yes. Let's keep it open.

Are you planning to implemnt in java the cddl part? Or you'll be happy to rely on the native library?
Cbor serialization is currently in pure Java only.
The native library dependency is only for address generation and transaction signing. But I am planning to remove that dependency in future release. That will simplify current testing and build cycle :)

Thanks.

Let's say I mint 1 token and instead of sending just 1.4 $ada (or thereabout) I want to directly send, let's say 10 $ada?

This is an interesting usecase. Unfortunately, the current high-level api (MintTransaction) doesn't have a "ada" value field.
Also, there is no high level api available to send both MintTransaction and PaymentTransaction in one transaction request.
Probably, a clean solution for future release :- Add a "value" field in MintTransaction to send any extra Ada in the mint transaction.
For now you have two options:
Option 1:
Use low level apis like TransactionInput / TransactionOutput. But in that case, you have to select utxos and build the Transaction in your application and sign using the provided api.
Option 2
Set min uxto value in TransactionDetailsParam to your required lovelace amount. Currently, default value is 1 Ada.
Pass the TransactionDetailsParam as part of mintToken().
Note: Currently, some extra ada value is getting added if you send a custom minAda. May be some unwanted side-effect due to change in min ada value in the client library which I need to debug.
But you can adjust the lovelace amount in TransactionDetailsParam based on your testing to avoid any extra ada value in the output.
Hope that helps.
Just wondering, if your project is public, would you be interested to add your project in the "Used By" section of this project.
MintTransaction paymentTransaction =
                MintTransaction.builder()
                        .sender(sender)
                        .receiver(receiver)
                        .mintAssets(Arrays.asList(multiAsset))
                        .policyScript(scriptPubkey)
                        .policyKeys(Arrays.asList(skey))
                        .build();

        TransactionDetailsParams detailsParams = TransactionDetailsParams.builder().ttl(getTtl()).build();
        detailsParams.setMinUtxoValue(new BigInteger("3000000"));

        BigInteger fee = feeCalculationService.calculateFee(paymentTransaction, detailsParams, metadata);
        paymentTransaction.setFee(fee);

        Result<TransactionResult> result = transactionHelperService.mintToken(paymentTransaction, detailsParams, metadata);

Suggestion number 1 seems to be working, but I can't seem to find a nice formula to map the desired amount to send to the minUtxo to set in the TransactionDetailsParams.
What I've found so far:

For 10 $ADA, minUtxo = 6.5
For 15 $ADA, minUtxo = 9.65
For 25 $ADA, minUtxo = 16.2

I'm happy to use this workaround, but I'm curious, if you know, where is the piece of code that is doing "weird" things, I might try to take a look. Thanks!

I have to still verify. But probably this behavior is expected as mentioned in the minAda calculation logic where minUtxo value is used. https://cardano-ledger.readthedocs.io/en/latest/explanations/min-utxo.html
If you want to check the implementation. you can check the following two files
MinAda calculation logic : https://github.com/bloxbean/cardano-client-lib/blob/85adffc45a2890228559e021fd16c14a4b3c3d9a/src/main/java/com/bloxbean/cardano/client/common/MinAdaCalculator.java#L9
Used here : https://github.com/bloxbean/cardano-client-lib/blob/85adffc45a2890228559e021fd16c14a4b3c3d9a/src/main/java/com/bloxbean/cardano/client/backend/api/helper/impl/UtxoTransactionBuilderImpl.java#L159
I will add the value attribute to MintTransaction in the next release,  then this workaround won't be required.

I will add the value attribute to MintTransaction in the next release, then this workaround won't be required.

<3


Add report-creator CLI tool

Introduce report-creator, a CLI tool that allows generating a JUnit test
report from the JSON output of the 'kubectl cluster-compare' plugin.
The tool categorizes the cluster comparison results into 3 test suites:

Diff Test Suite: Each test in this suite represents a matched CR compared
to a reference CR.

Missing CRs Test Suite: Tests in this suite represent CRs absent in the
cluster but present in the reference.

Unmatched CRs Test Suite: This suite includes all CRs in the cluster not
matched to any reference.

Additionally, tests of the new tool are added to ensure the the report creation
is as expected.  These tests use the same references used for the compare
command tests to validate the alignment of the reporter with the compare
command's reference and output format.

/hold
depends on #12

/unhold

/lgtm
/approve


Formatted code can be invalid Swift syntax

The following valid Swift code:
import SwiftUI
func foo() {
  @Environment(\.colorScheme) var colorScheme;
  print(colorScheme)
}

…gets formatted like this (notice the semi-colon is removed):
import SwiftUI
func foo() {
  @Environment(\.colorScheme) var colorScheme
  print(colorScheme)
}

This code is not valid Swift code. I'm not sure if this is a bug in the Swift parser, or perhaps this code cannot be unambiguously parsed (I filed a bug).
Should semi-colons be left in in order to work around this issue?

I'm definitely interested in knowing if this is intended behavior or not for the parser in general; thanks for filing the apple/swift issue.
But if the Swift compiler today requires the semicolon, then we'll have to make an exception to not remove it in that case.
Does this only happen for local variables with a property wrapper and without an explicit type annotation or initializer? Are there other cases that break in the same way?

Does this only happen for local variables with a property wrapper and without an explicit type annotation or initializer? Are there other cases that break in the same way?

Local variables with an inferred type is the only way we've come across it so far. Supplying an explicit type is the workaround we use when we come across this bug on CI.

@allevato This may have been fixed a little over a month ago (https://github.com/apple/swift/pull/61036), so maybe this won't be an issue in the next Swift release.

Tracked in Apple’s issue tracker as rdar://126948232


QoL improvement: detect_project_root() now works outside of repo tree

Main repo path determined from file and relative path instead of looking for pyproject.toml (which can break if code here is used in other repos).

Awesome, makes sense, thanks!!


fix(material/slider): some screen readers announcing long decimal values

Note: this is a resubmit of #20721 which had to be reverted because it was overriding people's aria-valuetext. This PR includes an input that allows for it to be changed.
It looks like some screen readers announce the value of a slider by calculating the percentage themselves using the aria-valuemin, aria-valuemax and aria-valuenow. The problem is that they don't round down the decimals so for a slider between 0 and 1 with a step of 0.1, they end up reading out values like 0.20000068. These changes work around the issue by setting aria-valuetext to the same value that we shown in the thumb which we truncate ourselves.
Fixes #20719.

Caretaker note: this PR includes an input to allow people to override the aria-valuetext, but some apps that bind it using [attr.aria-valuetext] might still fail. The apps would have to be changed to use <mat-slider [aria-valuetext]="..."> instead.


Improve the way to find tsconfig.json

I have a monorepo project, and its structure looks like this:
monorepo
├── packages
│  ├── packageA
│  └── packageB
└── index.mjs

I want to run the eslint command in the root directory, like:
const eslint = new ESLint({
    cwd: '/workspace/monorepo/packages/packageA',
    extensions: ['.ts', '.tsx', '.mts', '.cts'],
});

const result = await eslint.lintFiles('/workspace/monorepo/packages/packageA/index.ts');

console.log('result---', result);

The eslint-plugin-import plugin seems to be unable to find the tsconfig.json file of packageA:

I think eslint’s cwd configuration should be used first, it looks like this:
const tsconfigInfo = tsConfigLoader({
    cwd: context.parserOptions && context.parserOptions.tsconfigRootDir || eslint.options.cwd || process.cwd(),
    getEnv: (key) => process.env[key],
});

PS
At first I thought the context was the context configured by eslint, but later I found out that it was the plugin itself.

So I tried writing it like this and found that nothing worked 😅:
new ESLint({
    cwd: '/workspace/monorepo/packages/packageA',
    extensions: ['.ts', '.tsx', '.mts', '.cts'],
    overrideConfig: {
        parserOptions: {
            tsconfigRootDir: '/workspace/monorepo/packages/packageA'
         },
    },
})

Please stop doing this.

Duplicate of #274


Handle a spring resource properly because a resource may not be a file

Handle case where resource is not file

Well the whole idea of spring resource is that it an interface to a resource with different implementations. My particular implementation happens to be a type of object store in a semantic web database.. so when the service provider metadata is written it writes to that object store and generates an associated uri to point to it.

Garry

-------- Original message --------
From: Misagh Moayyed<EMAIL_ADDRESS>Date: 11/21/19 1:01 PM (GMT-05:00)
To: pac4j/pac4j<EMAIL_ADDRESS>Cc: garpinc<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [pac4j/pac4j] Handle a spring resource properly because a resource may not be a file (#1458)
@mmoayyed commented on this pull request.

In pac4j-saml/src/main/java/org/pac4j/saml/metadata/SAML2MetadataGenerator.javahttps://github.com/pac4j/pac4j/pull/1458#discussion_r349236230:

@@ -119,7 +118,7 @@
public MetadataResolver buildMetadataResolver(final Resource metadataResource) throws Exception {
final AbstractBatchMetadataResolver resolver;
if (metadataResource != null) {

       resolver = new FilesystemMetadataResolver(metadataResource.getFile());

       resolver = new SpringResourceMetadataResolver(metadataResource);

What is the use case here? Could you explain the scenario that has you define and load SP metadata that is not backed by a file? What specific implementation of the WritableResource are you dealing with?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/pac4j/pac4j/pull/1458?email_source=notifications&email_token=AAOW7RAQNDDMMKZ2X2NJ5H3QU3EITA5CNFSM4JP3DCG2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCMRUUFY#pullrequestreview-321079831, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAOW7RCL5WXDJMWE2D24TYDQU3EITANCNFSM4JP3DCGQ.

I'm hesitating on this: yes, the change is fairly minimal and would address all use cases, but do we really want to address such an unlikely use case?
@mmoayyed what do you think?

I don't think it's an unlikely use case that the file is not stored in a file system. In fact read does work but write currently doesn't. That's the whole reason we used resource in the first place.

Garry

-------- Original message --------
From: LELEU Jérôme<EMAIL_ADDRESS>Date: 11/22/19 3:02 AM (GMT-05:00)
To: pac4j/pac4j<EMAIL_ADDRESS>Cc: garpinc<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [pac4j/pac4j] Handle a spring resource properly because a resource may not be a file (#1458)
I'm hesitating on this: yes, the change is fairly minimal and would address all use cases, but do we really want to address such an unlikely use case?
@mmoayyedhttps://github.com/mmoayyed what do you think?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/pac4j/pac4j/pull/1458?email_source=notifications&email_token=AAOW7RALPZMU4P6KOK3INFDQU6GXZA5CNFSM4JP3DCG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE43YUQ#issuecomment-557431890, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAOW7RBUXJFHB5FMDVZO3YTQU6GXZANCNFSM4JP3DCGQ.

E.g. in my case: All data that is created during execution has to be stored in mongodb in gridFS, because all filesystems in production are read-only. So a GridFS backed implementation of WritableResource could do the trick. But when getFile is explicitly used, this cannot not work.

The reason the Spring resource class is used is primarily because we didn't want to re-create our own resource class, and not because we wanted to allow for alternative abstractions of resources; that is simply a side-benefit, but nothing to focus on. Furthermore, this is a fairly uncommon use case; as in, it's a lot more likely and common for an identity provider to want to store service metadata somewhere in a database, etc, than it is for a service provider to store its own metadata (or the identity provider's metadata) in a database. This is, for pac4j, demonstrated by the number of issues, pull requests and user activity in this area on the mailing list.
That said, I do like and support the idea of taking advantage of this abstraction but I am not a fan of introducing a whole new metadata-resolver resource by default with background threads and timers to refresh stuff, etc. That's complexity that is unwarranted at this point and is rather unnecessary until further evidence pops up. What would make sense to me, is allowing folks to define the metadata resource abstraction on their own as a pluggable strategy so that one can plug their own implementation into it and deal with object databases or anything else that is supported by that writeable-resource.
This pull request does a lot more and I think it needs to trim down first.

I submitted a PR that does it with injection instead.
From: Misagh<EMAIL_ADDRESS>Sent: Friday, November 22, 2019 7:40 AM
To<EMAIL_ADDRESS>Cc<EMAIL_ADDRESS><EMAIL_ADDRESS>Subject: Re: [pac4j/pac4j] Handle a spring resource properly because a resource may not be a file (#1458)
The reason the Spring resource class is used is primarily because we didn't want to re-create our own resource class, and not because we wanted to allow for alternative abstractions of resources; that is simply a side-benefit, but nothing to focus on. Furthermore, this is a fairly uncommon use case; as in, it's a lot more likely and common for an identity provider to want to store service metadata somewhere in a database, etc, than it is for a service provider to store its own metadata (or the identity provider's metadata) in a database. This is, for pac4j, demonstrated by the number of issues, pull requests and user activity in this area on the mailing list.
That said, I do like and support the idea of taking advantage of this abstraction but I am not a fan of introducing a whole new metadata-resolver resource by default with background threads and timers to refresh stuff, etc. That's complexity that is unwarranted at this point and is rather unnecessary until further evidence pops up. What would make sense to me, is allowing folks to define the metadata resource abstraction on their own as a pluggable strategy so that one can plug their own implementation into it and deal with object databases or anything else that is supported by that writeable-resource.
This pull request does a lot more and I think it needs to trim down first.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/pac4j/pac4j/pull/1458?email_source=notifications&email_token=AAOW7RAQALKI2WM4LSDDLATQU7HNJA5CNFSM4JP3DCG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE5QTLA#issuecomment-557517228, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAOW7RDVUT2VP7FREFKVOMDQU7HNJANCNFSM4JP3DCGQ.


ground work for custom scenarios

it's probably too much to implement custom scenarios all at one shot, but a good small step would be to let users upload their own geojson data and if it's in the appropriate format, add it to the list of scenarios we display.
maybe another bit would be to let the user download the resulting, merged, geojson data
the end game would, of course, be for some other process to generate this data, and then we can define an interface for that separately

see #45 for full discussion


Update Superposition.ipynb

Task 1.9. Superposition of all basis vectors.
Example doesn't correspond with the Goal

Also, I'm glad to see you've figured out how to send pull requests! :-)

Realised it was silly to not figure it out :-)
On Fri, 20 Nov, 2020, 2:02 AM Mariia Mykhailova<EMAIL_ADDRESS>wrote:

Also, I'm glad to see you've figured out how to send pull requests! :-)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/QuantumKatas/pull/567#issuecomment-730620469,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABK4O6BHU2PNCZMKZVC6HITSQV6EPANCNFSM4T3P5NQQ
.


Improve minification and add status for "FAILED_DEPENDENCY"

Reduces the min+gz from 1367 bytes to 1200 bytes - for those of use that use this library browser-side :) Also:

Fixed the status code for FAILED_DEPENDENCY. The status text was included, but the status code was missing.
Dropped the return null; since it's unreachable because of the throw.
throw an Error so there is a stack trace.

Ah Thanks zertosh. Nice catches. However I'm not sure it's worth trading the inconsequential 167 bytes for the awkward syntax. Any chance you could remove the minification aspect ?

@prettymuchbryce: Err, sorry, bad math. When you include the CJS wrapper, it goes from 3,242 to 2,260, shaving 982 bytes. Almost a full kb less - 30%. If you still feel the same way, I can go ahead and submit another PR just for the other changes.

After thinking about this a bit I think your PR is fine as is. The space savings is nice, and I don't think the readability is significantly impacted.

@prettymuchbryce Thanks! can you publish to npm too please?

Yep! Done.


BigQuery ingestion suceed but datasets not shown in UI

Hello,
I am encountering an issue with DataHub where BigQuery ingestion is reported as successful, but the BigQuery platform and datasets are not visible in the DataHub UI. Other platforms like MongoDB are ingested successfully and visible in the UI.

Issue Description:

BigQuery ingestion is reported as successful, and the connection test also succeeds.
The BigQuery platform and datasets are not visible in the DataHub UI.
Other platforms like MongoDB are visible in the UI after successful ingestion.

Environment:

DataHub version: 0.13.2
Helm chart version: datahub-0.4.19
Kubernetes version: 1.29
prerequisites: mysql ONLY, kafka and es are self hosted

Logs (some lines are omitted):

~~~~ Execution Summary - RUN_INGEST ~~~~
Execution finished successfully!
{'exec_id': '09f6853a-d4a2-4e39-a93e-6879ec25f7ca',
 'infos': ['2024-07-09 08:22:28.846333 INFO: Starting execution for task with name=RUN_INGEST',
           "2024-07-09 08:34:56.859026 INFO: Successfully executed 'datahub ingest'",
           '2024-07-09 08:34:56.859580 INFO: Finished execution for task with name=RUN_INGEST'],
 'errors': []}

{
  "cli": {
    "cli_version": "<IP_ADDRESS>",
    "py_version": "3.10.13",
    "os_details": "Linux-5.10.213-201.855.amzn2.x86_64-x86_64-with-glibc2.36",
    "mem_info": "396.41 MB",
    "peak_memory_usage": "396.41 MB",
    "disk_info": {
      "total": "53.67 GB",
      "used": "21.77 GB",
      "free": "31.9 GB"
    },
    "peak_disk_usage": "21.86 GB",
    "thread_count": 1,
    "peak_thread_count": 16
  },
  "source": {
    "type": "bigquery",
    "report": {
      "num_tables_fetch_sample_values_failed": 0,
      "num_tables_classification_attempted": 0,
      "num_tables_classification_failed": 0,
      "num_tables_classification_found": 0,
      "window_end_time": "2024-07-09 08:22:39.181089+00:00",
      "window_start_time": "2024-07-08 00:00:00+00:00",
      "events_produced": 14952,
      "events_produced_per_sec": 20,
      "entities": {
        "container": [ "sampled of 96 total elements"],
        "dataset": [ "sampled of 1559 total elements"],
        "tag": ["urn:li:tag:CLUSTERING_COLUMN_1", "urn:li:tag:CLUSTERING_COLUMN_2", "urn:li:tag:CLUSTERING_COLUMN_3", "urn:li:tag:CLUSTERING_COLUMN_4", "urn:li:tag:PARTITION_KEY"]
      },
      "aspects": {
        "container": {
          "containerProperties": 96,
          "status": 96,
          "dataPlatformInstance": 96,
          "subTypes": 96,
          "browsePathsV2": 96,
          "container": 95
        },
        "dataset": {
          "status": 1559,
          "schemaMetadata": 1545,
          "datasetProperties": 1545,
          "container": 1545,
          "dataPlatformInstance": 1545,
          "subTypes": 1545,
          "browsePathsV2": 1545,
          "viewProperties": 19,
          "datasetProfile": 1433,
          "operation": 1637,
          "datasetUsageStatistics": 275,
          "upstreamLineage": 179
        },
        "tag": {
          "tagKey": 5
        }
      },
      "warnings": {
        "profile skipped as partitioned table is empty or partition id or type was invalid": ["sampled of 93 total elements"]
      },
      "failures": {},
      "soft_deleted_stale_entities": ["sampled of 14 total elements"],
      "tables_scanned": 1526,
      "views_scanned": 19,
      "entities_profiled": 1433,
      "filtered": ["sampled of 370 total elements"],
      "num_view_definitions_parsed": 19,
      "num_view_definitions_failed_parsing": 0,
      "num_view_definitions_failed_column_parsing": 0,
      "view_definitions_parsing_failures": [],
      "profiling_skipped_not_updated": {},
      "profiling_skipped_size_limit": { "top 10 of total 30 entries": 97 },
      "profiling_skipped_row_limit": { "top 10 of total 22 entries": 40 },
      "profiling_skipped_other": {},
      "num_tables_not_eligible_profiling": {},
      "num_total_lineage_entries": {"<my-project>": 240},
      "num_skipped_lineage_entries_missing_data": {},
      "num_skipped_lineage_entries_not_allowed": {"<my-project>": 24},
      "num_lineage_entries_sql_parser_failure": {},
      "num_skipped_lineage_entries_other": {},
      "num_lineage_total_log_entries": {"<my-project>": 240},
      "num_lineage_parsed_log_entries": {"<my-project>": 240},
      "num_lineage_log_parse_failures": {},
      "lineage_failed_extraction": [],
      "lineage_metadata_entries": {"<my-project>": 163},
      "lineage_mem_size": {"<my-project>": "72.4 KB"},
      "lineage_extraction_sec": {"<my-project>": 24.77},
      "usage_extraction_sec": {"<my-project>": 43.97},
      "num_usage_total_log_entries": {"<my-project>": 21738},
      "num_usage_parsed_log_entries": {"<my-project>": 21721},
      "usage_error_count": {},
      "num_usage_resources_dropped": 368,
      "num_usage_operations_dropped": 149,
      "operation_dropped": [],
      "usage_failed_extraction": [],
      "num_project_datasets_to_scan": {"<my-project>": 97},
      "metadata_extraction_sec": { "top 10 of total 95 entries":<PHONE_NUMBER>9999998 },
      "include_table_lineage": true,
      "use_date_sharded_audit_log_tables": false,
      "log_page_size": 1000,
      "use_exported_bigquery_audit_metadata": false,
      "log_entry_start_time": "2024-07-09 07:08:59.489000+00:00",
      "log_entry_end_time": "2024-07-09 08:37:39.181089+00:00",
      "audit_start_time": "2024-07-08 23:45:00+00:00",
      "audit_end_time": "2024-07-09 08:37:39.181089+00:00",
      "upstream_lineage": {},
      "partition_info": {},
      "profile_table_selection_criteria": {},
      "selected_profile_tables": {},
      "profiling_skipped_invalid_partition_ids": {},
      "profiling_skipped_invalid_partition_type": {},
      "profiling_skipped_partition_profiling_disabled": [],
      "num_usage_workunits_emitted": 275,
      "total_query_log_entries": 0,
      "num_read_events": 10732,
      "num_query_events": 10989,
      "num_view_query_events": 9,
      "num_view_query_events_failed_sql_parsing": 0,
      "num_view_query_events_failed_table_identification": 0,
      "num_filtered_read_events": 17,
      "num_filtered_query_events": 0,
      "num_usage_query_hash_collisions": 0,
      "num_operational_stats_workunits_emitted": 1637,
      "snapshots_scanned": 0,
      "read_reasons_stat": {"JOB": 10732},
      "operation_types_stat": {
        "SELECT": 7823,
        "CREATE_TABLE_AS_SELECT": 1297,
        "MERGE": 2815,
        "DELETE": 28,
        "INSERT": 24,
        "ALTER_TABLE": 2,
        "CREATE_TABLE": 4,
        "TRUNCATE_TABLE": 2
      },
      "schema_api_perf": {
        "list_projects": "0.600 seconds",
        "list_datasets": "0.821 seconds",
        "get_columns_for_dataset": "70.436 seconds",
        "get_tables_for_dataset": "298.142 seconds",
        "list_tables": "10.798 seconds",
        "get_views_for_dataset": "105.928 seconds",
        "get_snapshots_for_dataset": "103.109 seconds"
      },
      "audit_log_api_perf": {
        "list_log_entries": "33.518 seconds"
      },
      "processing_perf": {
        "sql_parsing_sec": "0.552 seconds",
        "store_usage_event_sec": "6.489 seconds",
        "usage_state_size": "{'main': '44.1 MB', 'queries': '13.28 MB'}"
      },
      "lineage_start_time": "2024-07-09 07:23:59.489000+00:00",
      "lineage_end_time": "2024-07-09 08:22:39.181089+00:00",
      "stateful_lineage_ingestion_enabled": true,
      "usage_start_time": "2024-07-09 00:00:00+00:00",
      "usage_end_time": "2024-07-09 08:22:39.181089+
      "usage_end_time": "2024-07-09 08:22:39.181089+00:00",
      "stateful_usage_ingestion_enabled": true,
      "start_time": "2024-07-09 08:22:39.198077",
      "running_time": "12 minutes and 13.31 seconds"
    }
  },
  "sink": {
    "type": "datahub-rest",
    "report": {
      "total_records_written": 14952,
      "records_written_per_second": 20,
      "warnings": [],
      "failures": [],
      "start_time": "2024-07-09 08:22:32.441719",
      "current_time": "2024-07-09 08:34:52.516074",
      "total_duration_in_seconds": 740.07,
      "max_threads": 15,
      "gms_version": "v0.10.4",
      "pending_requests": 0
    }
  }
}

[earlier logs truncated...]
[2024-07-09 08:32:55,675] INFO     {datahub.ingestion.source.bigquery_v2.bigquery:711} - Starting profiling project <my-project>
[2024-07-09 08:32:55,676] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_report:160} - Time spent in stage <<my-project>: Metadata Extraction at 2024-07-09 08:22:39.951665+00:00>: 615.72 seconds

[2024-07-09 08:33:03,712] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_report:160} - Time spent in stage <<my-project>: Profiling at 2024-07-09 08:32:55.676658+00:00>: 8.04 seconds
[2024-07-09 08:33:03,774] INFO     {datahub.ingestion.source.bigquery_v2.usage:880} - Start loading log entries from BigQuery for <my-project> with start_time=2024-07-08 23:45:00+00:00 and end_time=2024-07-09 08:37:39.181089+00:00
[2024-07-09 08:33:06,900] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 0 log entries from GCP Log for <my-project>
[2024-07-09 08:33:07,189] WARNING  {py.warnings:109} - /tmp/datahub/ingest/venv-bigquery-2b9c1ab97dc6cd7f/lib/python3.10/site-packages/datahub/ingestion/source/bigquery_v2/bigquery_audit.py:185: DeprecationWarning: Call to deprecated class method create_from_string. (prefer .from_string)
  dataset_urn = DatasetUrn.create_from_string(urn)

[2024-07-09 08:33:07,191] WARNING  {py.warnings:109} - /tmp/datahub/ingest/venv-bigquery-2b9c1ab97dc6cd7f/lib/python3.10/site-packages/datahub/ingestion/source/bigquery_v2/bigquery_audit.py:186: DeprecationWarning: Call to deprecated method get_dataset_name. (Use .name instead) -- Deprecated since version <IP_ADDRESS>.
  split = dataset_urn.get_dataset_name().rsplit(".", 3)

[2024-07-09 08:33:08,961] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 1000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:10,684] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 2000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:12,679] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 3000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:14,632] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 4000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:16,618] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 5000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:18,881] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 6000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:20,909] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 7000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:22,892] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 8000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:24,381] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 9000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:25,898] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 10000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:27,540] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 11000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:29,629] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 12000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:31,660] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 13000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:33,618] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 14000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:35,748] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 15000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:37,759] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 16000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:39,464] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 17000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:41,468] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 18000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:44,196] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 19000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:45,851] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 20000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:47,391] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 21000 log entries from GCP Log for <my-project>
[2024-07-09 08:33:47,678] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:137} - Finished loading log entries from GCP Log for <my-project>
[2024-07-09 08:33:47,679] INFO     {datahub.ingestion.source.bigquery_v2.usage:483} - Total number of events aggregated = 21345.
[2024-07-09 08:33:47,679] INFO     {datahub.ingestion.source.bigquery_v2.usage:486} - Total number of read events generated = 0.
[2024-07-09 08:33:48,300] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_report:160} - Time spent in stage <<my-project>: Usage Extraction Ingestion at 2024-07-09 08:33:03.713020+00:00>: 44.59 seconds

[2024-07-09 08:34:04,271] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_report:160} - Time spent in stage <*: Usage Extraction Operational Stats at 2024-07-09 08:33:48.300941+00:00>: 15.97 seconds
[2024-07-09 08:34:11,073] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_report:160} - Time spent in stage <*: Usage Extraction Usage Aggregation at 2024-07-09 08:34:04.271522+00:00>: 6.8 seconds
[2024-07-09 08:34:11,074] INFO     {datahub.ingestion.source.bigquery_v2.lineage:337} - Generate lineage for <my-project>
[2024-07-09 08:34:11,074] INFO     {datahub.ingestion.source.bigquery_v2.lineage:713} - Entering create lineage map function
[2024-07-09 08:34:11,075] INFO     {datahub.ingestion.source.bigquery_v2.lineage:640} - Populating lineage info via exported GCP audit logs
[2024-07-09 08:34:11,140] INFO     {datahub.ingestion.source.bigquery_v2.lineage:643} - Start loading log entries from BigQuery for <my-project> with start_time=2024-07-09 07:08:59.489000+00:00 and end_time=2024-07-09 08:37:39.181089+00:00
[2024-07-09 08:34:13,271] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:126} - Loaded 0 log entries from GCP Log for <my-project>
[2024-07-09 08:34:35,839] INFO     {datahub.ingestion.source.bigquery_v2.bigquery_audit_log_api:137} - Finished loading log entries from GCP Log for <my-project>
[2024-07-09 08:34:35,839] INFO     {datahub.ingestion.source.bigquery_v2.lineage:798} - Exiting create lineage map function
[2024-07-09 08:34:35,839] INFO     {datahub.ingestion.source.bigquery_v2.lineage:358} - Built lineage map containing 163 entries.
[2024-07-09 08:34:36,174] INFO     {datahub.ingestion.source.state.checkpoint:145} - Successfully constructed last checkpoint state for job ingest_from_bigquery_source with timestamp 2024-07-09 07:46:57.329000+00:00
[2024-07-09 08:34:36,175] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,176] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,184] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,207] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,216] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,226] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,231] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,231] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,231] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,232] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,232] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,232] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,233] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:36,233] INFO     {datahub.ingestion.source.state.stale_entity_removal_handler:180} - Soft-deleting stale entity - urn:li:dataset:(urn:li:dataPlatform:bigquery,<my-project>....,PROD)
[2024-07-09 08:34:49,132] INFO     {datahub.ingestion.run.pipeline:504} - Processing commit request for DatahubIngestionCheckpointingProvider. Commit policy = CommitPolicy.ON_NO_ERRORS, has_errors=False, has_warnings=True
[2024-07-09 08:34:51,911] INFO     {datahub.ingestion.run.pipeline:524} - Successfully committed changes for DatahubIngestionCheckpointingProvider.
[2024-07-09 08:34:51,916] INFO     {datahub.ingestion.reporting.file_reporter:52} - Wrote SUCCESS report successfully to <_io.TextIOWrapper name='/tmp/datahub/ingest/09f6853a-d4a2-4e39-a93e-6879ec25f7ca/ingestion_report.json' mode='w' encoding='UTF-8'>
[2024-07-09 08:34:51,917] INFO     {datahub.cli.ingest_cli:141} - Finished metadata ingestion

-ScreenShots:

Although it is currently running during the re-execution attempt, the logs indicate that it has successfully completed

Only Mongodb Platform Shown

I would appreciate any advice on how to resolve this issue.
Thank you.

Please, can you check the ingestion summary in the logs?
It should show you how many dataset were ingested and also it should show you some sample dataset which was ingested.

I resolved this issue (but I don't know why) so closed it.


Concat dependency

Hi,
I'm using librarian puppet and am getting a conflict between the dependencies of this module and other modules in use.
The dependency is:-
{ "name": "puppetlabs/concat", "version_requirement": ">=1.0.0 <=1.2.1" }
and I have some other modules requiring concat >=1.2.5.
I'm off to test locally but is there really a hard requirement on this version of concat?
Thanks.

Hopefully not but we do seem to have a good reason in https://github.com/gds-operations/puppet-auditd/commit/9a6d8cac16541d35ce23d2f190d213e54685122f
We would welcome a PR which bumped it though.

I'm not much of a coder but I've had a go.

Hi,
Sorry to be a pain but any chance you could have a quick look at the MR?
Thanks.

This is hopefully fixed in master now that #39 has been merged so I'm going to close this issue. Thank you for raising the problem and (especially) the PR to fix it.


Update verifiable-credentials-configure-tenant.md

Add clarity on granting admin consent for the API Permissions. I see the message this repo doesn't allow adding file so the image URL will need to be changed to (media/verifiable-credentials-configure-tenant/granted-admin-consent-app-api-permissions-verifiable-credentials.png) from (https://user-images.githubusercontent.com/28261714/151245893-5b104e0f-a21d-4542-b6a7-de3c94fa5416.png) after uploading the image there.

@premgan : Thanks for your contribution! The author(s) have been notified to review your proposed change.

I see the message this repo doesn't allow adding file.

Hi @premgan - The repository accepts image files, but the github.com pull request UI does not. To include an image file in your pull request, you'd need to use the tools that are described in Install content authoring tools.

@premgan thank you for your feedback. Please submit your changes following the instructions that @ktoliver shared.

#please-close


Would This work with ADAFRUIT FEATHER M0 WIFI - ATSAMD21 + ATWINC1500?

I'm really new to embedded so keep that in mind.
Would this be able to make use of this board https://www.adafruit.com/product/3010 and make use of the wifi?
Thanks!

I'm not aware of a rust crate for the ATWINC1500.

@fionawhim is working on wifi support for boards like the pyportal that use esp32 for wifi https://github.com/fionawhim/wifinina

Yep. Unfortunately the ESP32 WiFiNINA looks to share nothing with the ATWINC1500 besides both being communicated with over SPI. (From the look of it, Adafruit indicates that driving a ATWINC1500 is so complicated that they can’t fit it in Python on an M0?)
My stuff will work with PyPortal and, presumably, anything that Adafruit markets as "AirLift."

Thanks for the quick response @sajattack and @fionawhim


Add zip support to the dump command

What does this PR change? What problem does it solve?
This PR loosely based on @brentadamson PR #2433, the main difference is splitting walk and tar/zip logic.
I also added some tests (copy-pasted from tar part) and fix some bugs.
Was the change discussed in an issue or in the forum before?
https://forum.restic.net/t/restoring-files-directly-into-a-single-compressed-archive/968/13
Checklist

[x] I have read the Contribution Guidelines
[x] I have enabled maintainer edits for this PR
[x] I have added tests for all changes in this PR
[x] I have added documentation for the changes (in the manual)
[x] There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
[x] I have run gofmt on the code in all commits
[x] All commit messages are formatted in the same style as the other commits in the repo
[x] I'm done, this Pull Request is ready for review

I would be grateful if someone execute go test -v -run TestWriteZip on Linux and post the results.

@DRON-666:

I would be grateful if someone execute go test -v -run TestWriteZip on Linux and post the results.

That's the only output I see after the test ran for ~3 minutes:
?   	github.com/restic/restic	[no test files]

Thank you and I'm sorry that I was not too specific: you should run it from <restic repo>/internal/dump/ or use go test -v -run TestWriteZip github.com/restic/restic/internal/dump instead.

go: downloading github.com/minio/sha256-simd v0.1.1
go: downloading github.com/pkg/xattr v0.4.1
go: downloading github.com/restic/chunker v0.4.0
go: downloading golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
go: downloading golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
go: downloading github.com/pkg/errors v0.9.1
go: downloading gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637
go: downloading github.com/cenkalti/backoff/v4 v4.0.2
go: downloading github.com/dchest/siphash v1.2.2
go: downloading github.com/elithrar/simple-scrypt v1.3.0
go: downloading golang.org/x/sys v0.0.0-20201007082116-8445cc04cbdf
=== RUN   TestWriteZip
=== RUN   TestWriteZip/single_file_in_root
    testing.go:30: using low-security KDF parameters for test
    config.go:65: disabling check of the chunker polynomial
    zip_test.go:74: chdir to /tmp/restic-test-319287848
    zip_test.go:149: Mode is -rw-r----- [000001a0] for file
    zip_test.go:90: chdir back to /home/alex/Github/restic/restic/internal/dump
=== RUN   TestWriteZip/multiple_files_in_root
    testing.go:30: using low-security KDF parameters for test
    config.go:65: disabling check of the chunker polynomial
    zip_test.go:74: chdir to /tmp/restic-test-671586865
    zip_test.go:149: Mode is -rw-r----- [000001a0] for file1
    zip_test.go:149: Mode is -rw-r----- [000001a0] for file2
    zip_test.go:90: chdir back to /home/alex/Github/restic/restic/internal/dump
=== RUN   TestWriteZip/multiple_files_and_folders_in_root
    testing.go:30: using low-security KDF parameters for test
    config.go:65: disabling check of the chunker polynomial
    zip_test.go:74: chdir to /tmp/restic-test-402272110
    zip_test.go:149: Mode is -rw-r----- [000001a0] for file1
    zip_test.go:149: Mode is -rw-r----- [000001a0] for file2
    zip_test.go:149: Mode is drwxr-x--- [800001e8] for firstDir/
    zip_test.go:149: Mode is -rw-r----- [000001a0] for firstDir/another
    zip_test.go:149: Mode is drwxr-x--- [800001e8] for secondDir/
    zip_test.go:149: Mode is -rw-r----- [000001a0] for secondDir/another2
    zip_test.go:90: chdir back to /home/alex/Github/restic/restic/internal/dump
=== RUN   TestWriteZip/file_and_symlink_in_root
    testing.go:30: using low-security KDF parameters for test
    config.go:65: disabling check of the chunker polynomial
    zip_test.go:74: chdir to /tmp/restic-test-459097730
    zip_test.go:149: Mode is -rw-r----- [000001a0] for file1
    zip_test.go:149: Mode is Lrwxrwxrwx [080001ff] for file2
    zip_test.go:90: chdir back to /home/alex/Github/restic/restic/internal/dump
--- PASS: TestWriteZip (0.02s)
    --- PASS: TestWriteZip/single_file_in_root (0.00s)
    --- PASS: TestWriteZip/multiple_files_in_root (0.00s)
    --- PASS: TestWriteZip/multiple_files_and_folders_in_root (0.00s)
    --- PASS: TestWriteZip/file_and_symlink_in_root (0.01s)
PASS
ok  	github.com/restic/restic/internal/dump	0.028s


Implement PartialOrd and Ord traits

I am attempting to use the FourCC values in a match statement, the compiler said that PartialOrd and Ord traits are both necessary.
Implementing something basic here. What do you think?

Hey @dholroyd, do you have any thoughts on this one?


[BEAM-2899]  Fork FnDataService from runners-core

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Also remove the runners-core fn package.


Does SWA support writing files to a tmp directory?

I'm currently trying to deploy a Next.js app that writes stuff to the tmp directory using: https://github.com/stefanpenner/sync-disk-cache
This is used by the Sitecore JSS next.js SDK:
https://github.com/Sitecore/jss/blob/dev/packages/sitecore-jss-nextjs/src/editing/editing-data-cache.ts
The editing data cache is currently getting mis-hits, I assume because the SWA is not allowed to write to the tmp directory?

So my question is, is this even supported? Or do I need to provide a different os.tmpdir?

@erwinsmit  "Editing data cache miss for key" means cache didn't have the key looks like. Do you see any errors when trying to set the cache, as that's when the write to tmp folder happens, and maybe see if there are any logs related to that.
Also, we use consumption plan functions which can scale based on traffic and I think the tmp data written might not be persisted for longer durations or across the calls.

I couldn't find any errors specifically to writing the file. I re-wrote this bit of functionality to use blobstorage instead, this seems to work better in this case. Especially if the are issues with scaling and losing persisted data.
Does that mean that when the functions scale based on traffic, the TMP directory is bound to each scaled up instance?


kedge init should not allow non int values for port

$ kedge init -n batman -i batcave -p gotham
file kedge.yml created

$ cat kedge.yml
name: batman
containers:
- image: batcave
services:
- ports:
  - port: gotham

ping @surajnarwade

closed in #232


CodeActAgent: Only delegate to BrowsingAgent as last resort

#2103 was checked-in to main branch yesterday, but it is reported that it seems to be an overkill (and waste at least one round of conversation) to delegate a simple web browsing action to BrowsingAgent.
This PR is a partial revert of the previous PR: it brings browsing action back, and prompts CodeActAgent to delegate to BrowsingAgent only if it cannot find any other way. Hopefully this will also make LLM to browse less - it is also reported that CodeActAgent likes to browse even if it could just use API.
This PR doesn't solve one problem: sometimes the browsing agent has finished its job and reported back the result, yet codeact agent still decides to try other ways or delegates again, as if the job is not done. My guesstimate is since we only include the final observation of child agent in parent agent's history, the parent loses the chain of thought to achieve the result, and thus gets confused. I believe the solution would be to append child's history to parent's history, but in a summary format (a.k.a. "condensed"). @enyst has been doing some work and preliminary result seems promising. See discussion here https://github.com/OpenDevin/OpenDevin/pull/2103#discussion_r1631615417.
I quickly tried the browsing integration test. Now CodeActAgent would first use simple goto browse action, figures out the answer is hidden behind a button, and then decides to delegate to BrowsingAgent. This seems promising, but it might also just because I included a similar example in the one-shot learning.

This looks OK, but it makes the prompt more complex. It'd be good to evaluate this against SWE-bench lite to check its effect on accuracy.

I believe this will lower the score - the performance between CodeActAgent and CodeActSWEAgent on SWE-bench lite has proved so. Question is how much it hurts the benchmark - that would be interesting to know.

This looks OK, but it makes the prompt more complex. It'd be good to evaluate this against SWE-bench lite to check its effect on accuracy.

I believe this will lower the score - the performance between CodeActAgent and CodeActSWEAgent on SWE-bench lite has proved so. Question is how much it hurts the benchmark - that would be interesting to know.

I think CodeActAgent for end-user, and CodeActSWEAgent for benchmark. So CodeActAgent can explore on multi-agent mode( maybe some other new skills/modes ), and don't pay too much attention on the benchmark score, and  CodeActSWEAgent focus on the benchmark score, accuracy/cost/performance are the most important, so maybe new skills/modes won't used at the beginning.  @li-boxuan @neubig

So CodeActAgent can explore on multi-agent mode( maybe some other new skills/modes ), and don't pay too much attention on the benchmark score

Yeah I mostly agree. That being said, I guess it's still interesting to see how CodeActAgent perform differently than CodeActSWEAgent on SWE-bench lite though.

Converted to a draft since this is not ready for review at the moment.

I am closing this PR. I feel like this makes our prompt unnecessarily complicated. Just simply delegating browsing actions to BrowsingAgent seems to be the right thing to do.
FWIW, one of the motivation of this PR,

This PR doesn't solve one problem: sometimes the browsing agent has finished its job and reported back the result, yet codeact agent still decides to try other ways or delegates again, as if the job is not done

this bug was fixed.


Problem getting vector (array) of strings

Considering the JSON below,
{
     "names": ["Tim", "Tom"],
     "num": [1, 2]
}

I want to get a vector of string. I realized that I could only use auto a = json["name"] and then process it using for-range loop. However, if I want to explicitly define the return type then I will get an error. For instance, both vector<string> names = j["names"] and vector<string> names = j["names"].get<vector<string>>() will give me an error:
./json.hpp:829:24: error: no matching constructor for initialization of 'std::__1::vector<std::__1::basic_string<char>,
      std::__1::allocator<std::__1::basic_string<char> > >'
                return T(m_value.array->begin(), m_value.array->end());

I can see that json library couldn't cast to vector<string>; however, it will work if I want to get num even explicitly like this: vector<float> f = j["num"].get<vector<float>>().
Now, I'm wondering if I am doing it wrong or it's not implemented?
Thanks,
Amir.

Dear Amir,
thanks for reporting! It sure looks like a bug. I shall have a look once I'm back from holidays.
All the best
Niels

I could reproduce the bug.

Unfortunately, I cannot find a solution right now. I'll check again later.

The last commit (1bdb6acb1fb003a9bb8cb7771618f1630e78f7d0) fixed this issue. The following code compiles as expected:
#include <src/json.hpp>

using nlohmann::json;

int main()
{
    auto j = R"(
    {
         "names": ["Tim", "Tom"],
         "num": [1, 2]
    }
    )"_json;

    std::vector<std::string> names = j["names"];

    for (auto i : names)
    {
        std::cout << i << " ";
    }

    // prints: Tim Tom
}

Thanks! :)

#include <nlohmann/json.hpp>
#include <iostream>

using nlohmann::json;

int main() {
  auto j = R"(
    {
         "names": ["Tim", "Tom"],
         "num": [1, 2]
    }
    )"_json;

  std::vector<std::string> names;
  names = j["names"];

  for (auto i : names) {
    std::cout << i << " ";
  }

  // prints: Tim Tom
}

main.cpp:14:20: error: ambiguous overload for ‘operator=’ (operand types are ‘std::vector<std::__cxx11::basic_string >’ and ‘nlohmann::basic_json<>::value_type {aka nlohmann::basic_json<>}’)
names = j["names"];

Hi @nlohmann , separating the definition and declaration doesnt work again. Any solution for this? I need this.

You modified the code slightly and that might affect your code. I think you should either use the names = j.at("names") or indicates the types in get again, names = j.get<std::vector<std:string>>().


Investigate removing hard-coded dependency on node_modules

See the yarnrc.yml and Podfile in the example project. It looks like the current approach is to hard-code path dependencies, which feels like a code smell to me, and further assumes that the end user will always be using npm or some other package manager in a legacy compatibility mode, and this is suboptimal at best.

Tagging with help-wanted since I almost assuredly lack the knowledge to know if this is even a reasonable request. It's also pretty baked as an android assumption too.

I'm not sure react-native has good support for PNP installs right now. Will take a closer look.

See react-native-community/cli#27 for some context. PnP support looks like is not implemented in react-native and not a priority at this time.
I just tried to bootstrap a react native app with react native 0.70.6 and found the same hard-coded references to node_modules in ios Podfile, but also in android build.gradle. See in the example app:
https://github.com/maplibre/maplibre-react-native/blob/e6dffb21c34648ad4924ccd6ed5d22fe2a1fe38e/example/android/build.gradle#L29
So in my opinion there isn't much we can do to remove it at this time.

Yeah, makes sense. Thanks a bunch for the research on this.
I've been able to bootstrap projects that install the maplibre-react-native package locally (via yalc) just fine, as well as the examples, so it's probably not an issue using pnp for the development of the module itself as it's all packaged the same at the end of the day (IIUC), but it looks like we can't get around the hard dependency on a node_modules folder for actual apps at the end of the day. And I guess that's not a huge issue since nobody except me really cares ;)


Include enums in Bond runtime schema

Now enums are included in schema as int32. Dynamic parsers might need a way to resolve int32 to enum value as a string.
I suggest to include it in the runtime schema like this:
"structs": [
 {
  "metadata": {
     "name": "SomeStruct"
   },
  "fields": [
   {
      "metadata": {
         "name": "someField"
       },
      "id" : 1,
      "type": {
          "id": 16,
          "struct_def": 0,
          "enum_def": 7    -- this is new field that references Enum. Or we can reuse "struct_def" when type id is 16
       },
   },
  ]
 }
],
"enums": [      -- new top level node with all Enums
  {
    "metadata": {
         "name": "Direction",
    },
    "items": [
        {
            "item": "LEFT",
            "id": 1,
        },
        {
            "item": "RIGHT",
            "id": 2,
        },
    ]
  }
]

This is a good idea, but this is not a change the Bond project wants to make now or in the foreseeable future, as it has a high potential to break existing, deployed applications. I'm going to close this as "Won't Fix".
Changes to Bond's runtime schema--even additive ones like this that should be a non-breaking change--are basically impossible to make without breaking existing applications. There are, unfortunately, many applications that have serialized runtime schema data in Simple Binary and these applications have no way to deal with any schema changes to runtime schemas. (For example, see the Bond 5.0.0 release where we had to back out an additive change to runtime schemas.)


Update README.md

Update
I updated the readme file .. Please review and accept

Thoughts
I probably shouldn't but okay :1234:


Create a video for: /layouts/index.html

Gives a brief introduction and overview of the layouts section that describes all the principles at a high-level.

video added to rwd-basics, and udacity course


fix(store/file): wait for q4 to be written before returning

Fixed a race condition when writing/removing the Q4 file. The main go-routine wasn’t waiting for the Q4 write to finish, causing the file to be removed before the write completed. This could leave unpredictable data on disk after the removal. These changes ensure the write operation finishes before any removal is attempted.

Codecov Report
Attention: Patch coverage is 30.00000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 44.80%. Comparing base (2469e7a) to head (0491bb2).
Report is 349 commits behind head on main.

Files with missing lines
Patch %
Lines

store/file/ods_q4.go
30.00%
5 Missing and 2 partials :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3865      +/-   ##
==========================================
- Coverage   44.83%   44.80%   -0.03%
==========================================
  Files         265      314      +49
  Lines       14620    21804    +7184
==========================================
+ Hits         6555     9770    +3215
- Misses       7313    11000    +3687
- Partials      752     1034     +282

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Drop use_2to3

It's no longer supported by setuptools.

Tested by changing the SHA in requirements.txt in Memrise and running ➜ docker compose build memrise locally.

https://memrise.atlassian.net/browse/BA-7176


Made the first heading in each docs page the page title

Accomplished using the same regex that generates the keyword index page. I did not personally check all 9,125 docs files that changed, so merge with caution. Also adds the ability to give a title prefix to a page (these need to be translated for all of the languages in the docs)

Merged. Feel free to close!


Broken with git merge.conflictstyle zdiff3

Hi,
I'm using zdiff3 as diff style in my ~/.gitconfig:
[merge]
	conflictStyle = zdiff3

This feature was introduced in Git 2.35.
When trying to generate a project it just hard fails:
cargo generate --git https://github.com/rustwasm/wasm-pack-template
⚠   Unable to load config file: $HOME/.cargo/cargo-generate.toml
Error: ⛔   Git Error: unknown style 'zdiff3' given for 'merge.conflictstyle'; class=Checkout (20)

Environment:
cargo-generate 0.13.0
cargo 1.59.0 (49d8809dc 2022-02-10)
rustc 1.59.0 (9d1b2106e 2022-02-23)

it probably depends on https://github.com/rust-lang/git2-rs/issues/795

The rust-lang/git2-rs#795 has now been closed, waiting for update to  git2 dependecy, then we need to retest.

@Nothing4You could you please retest with the newest version - it's been long fixed bi the git2 dependecy, so???