Intermitten System.Net.Sockets.SocketException followed by System.Net.Http.HttpRequestException when using S3 to upload and download many files.

I am experiencing an intermittent exception when downloading or uploading files to S3. My function is using a single instance of Amazon.S3.AmazonS3Client to both download and upload files. The downloads are limited to 5 at once and the uploads are limited to 5 at once for a total of 10. The configuration is unchanged from the default except for raising the BufferSize to 1 MB.
When the function starts it always works for a couple of minutes until it eventually throws the following exception (from either awaiting the result on a download or upload request):
System.Net.Http.HttpRequestException: No such host is known ---> System.Net.Sockets.SocketException: No such host is known
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\_mobile\HttpRequestMessageFactory.cs:line 524
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 175
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\RedirectHandler.cs:line 59
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\Unmarshaller.cs:line 85
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Services\S3\Custom\Internal\AmazonS3ResponseHandler.cs:line 58
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 99
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 61
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\EndpointDiscoveryHandler.cs:line 75
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\EndpointDiscoveryHandler.cs:line 86
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CredentialsRetriever.cs:line 98
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RetryHandler\RetryHandler.cs:line 122
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RetryHandler\RetryHandler.cs:line 138
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 61
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 61
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Services\S3\Custom\Internal\AmazonS3ExceptionHandler.cs:line 57
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\ErrorCallbackHandler.cs:line 53
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\MetricsHandler.cs:line 65

I'm currently running this instance from the 'AWS .NET Mock Lambda Test Tool' but I've seen similar issues (but much less frequent) when deployed.
Expected Behavior
If I'm hitting some kind of request limit and S3 is booting my connection, then I'd expect there to be some kind of way for the client to tell.
Current Behavior
Intermittently one of my downloads or uploads throws the above exception, which as the end client I'm not sure I should be handling.
I've turned on the SDK debugging using:
AWSConfigs.LoggingConfig.LogTo = LoggingOptions.SystemDiagnostics;
AWSConfigs.LoggingConfig.LogMetrics = true;
AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.OnError;
and attached the output right before the issue:
DebugOutput.txt
It always starts with a socket exception that I haven't been able to inspect yet since it gets swallowed up by the retry mechanism:
Amazon Information: 5863 : Exception thrown: 'System.Net.Sockets.SocketException' in System.Private.CoreLib.dll
Possible Solution

Steps to Reproduce (for bugs)
This is an icky bug because I don't know how to simplify my problem but cannot release all of my code for proprietary reasons. Here's the essentials of what the code is doing:
public class S3Iterator
{
	public S3Iterator(IAmazonS3 in_s3Client, string in_bucket, List<string> in_keys, int in_maxSimultaneousDownloads)
	{
		m_s3Client = in_s3Client;
		m_bucket = in_bucket;
		m_keys = in_keys;
		m_maxSimultaneousDownloads = in_maxSimultaneousDownloads;
	}

	// The client we use to download objects from S3.
	protected IAmazonS3 m_s3Client;

	// The bucket we are downloading from in S3.
	protected string m_bucket;

	// The list of keys for this iterator.
	protected List<string> m_keys;

	// The maximum amount of files we'll download at once.
	protected int m_maxSimultaneousDownloads;
}

/// Helper class that is used to download multiple keys from S3 asynchronously and iterate over the results.
/// Using this class makes it easy to process files while downloading others in the background. The order that
/// the responses come back is not guaranteed to be the same order as the keys passed in. Instead we return the
/// first file to be finished to minimize the amount of time spent blocking.
public class GetObjectsNonSequentialIterator : S3Iterator, IEnumerable<Amazon.S3.Model.GetObjectResponse>
{
	public GetObjectsNonSequentialIterator(IAmazonS3 in_s3Client, string in_bucket, List<string> in_keys, int in_maxSimultaneousDownloads = 5) :
		base(in_s3Client, in_bucket, in_keys, in_maxSimultaneousDownloads)
	{
		// Immediately start downloading the first few items instead of waiting for the first call to
		// 'GetEnumerator'.
		FillGetObjectResponsesList();
	}

	public IEnumerator<Amazon.S3.Model.GetObjectResponse> GetEnumerator()
	{
		while (m_getObjectResponses.Count > 0)
		{
			// Find the index of the first download completed.
			var indexOfFinishedTask = m_getObjectResponses.FindIndex((in_task) => in_task.IsCompleted);

			// No download is finished so force a wait.
			if (indexOfFinishedTask == -1)
			{
				Task.WaitAny(m_getObjectResponses.ToArray());

				// At this point we are guaranteed at least one item is complete.
				indexOfFinishedTask = m_getObjectResponses.FindIndex((in_task) => in_task.IsCompleted);
			}

			// Get the item to return and remove it from the list.
			var getObjectResponse = m_getObjectResponses[indexOfFinishedTask].Result;
			m_getObjectResponses.RemoveAt(indexOfFinishedTask);

			// Queue up any additional downloads needed.
			FillGetObjectResponsesList();

			// And return the item.
			yield return getObjectResponse;
		}
	}

	System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
	{
		return GetEnumerator();
	}

	/// <summary>
	/// Fills m_getObjectResponses until it either reaches m_maxSimultaneousDownloads or we run out of keys to
	/// download.
	/// </summary>
	private void FillGetObjectResponsesList()
	{
		while (
			m_getObjectResponses.Count < m_maxSimultaneousDownloads &&
			m_nextKeyIndexToGet < m_keys.Count)
		{
			m_getObjectResponses.Add(m_s3Client.GetObjectAsync(new Amazon.S3.Model.GetObjectRequest()
			{
				BucketName = m_bucket,
				Key = m_keys[m_nextKeyIndexToGet]
			}));
			++m_nextKeyIndexToGet;
		}
	}

	// The index into m_keys we should download next.
	private int m_nextKeyIndexToGet = 0;

	// The responses to our get object requests.
	private List<Task<Amazon.S3.Model.GetObjectResponse>> m_getObjectResponses = new List<Task<Amazon.S3.Model.GetObjectResponse>>();
};

var s3Client = new Amazon.S3.AmazonS3Client();
var putObjectTasks = new List<Task>();

For each Foo:
{
   var s3SourceKeys = s3Client.ListObjectsV2Async(Request To Get Foo's Keys).Wait();
   var getObjectsIterator = new GetObjectsNonSequentialIterator(s3Client, MyLib.AWS.S3.Buckets.MyBucket, s3SourceKeys);
   foreach (var response in getObjectsIterator)
   {
       <Process, I use the response and then dispose of it>
   }

   For each Bar:
   {
      <Process>
      putObjectTasks.Add(s3Client.PutObjectAsync(RequestUsingStream));
     <LimitConcurrentPutObjectTasksTo5>
   }
}

Context
I'm trying to download data, process it, and upload it in an efficient manner. I've been searching for best practices and it is very possible that I'm doing something wrong but I'm completely stumped at this point.
Your Environment

AWSSDK.Core version used: <IP_ADDRESS>
Service assembly and version used:
Operating System and version: Microsoft Windows 10 Pro 10.0.17134 Build 17134
Visual Studio version: Visual Studio 2017 Professional
Targeted .NET platform: NETCore 2.1

.NET Core Info

.NET Core version used for development: 2.1
.NET Core version installed in the environment where application runs: 2.1
Output of dotnet --info:
.NET Core SDK (reflecting any global.json):
Version:   2.1.505
Commit:    b220231234

Runtime Environment:
OS Name:     Windows
OS Version:  10.0.17134
OS Platform: Windows
RID:         win10-x64
Base Path:   C:\Program Files\dotnet\sdk\2.1.505\
Host (useful for support):
Version: 2.1.9
Commit:  dcedc87d22
.NET Core SDKs installed:
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.401 [C:\Program Files\dotnet\sdk]
2.1.403 [C:\Program Files\dotnet\sdk]
2.1.505 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

Contents of project.json/project.csproj:

netcoreapp2.1
true
Lambda

Hi @helbize, could you try out our Transfer Utility and see if that meets your needs?  It might be easier and less problematic to use that.  You may also want to ensure that you have set a sufficient timeout and have a reliable internet connection with available bandwidth.

Interestingly, I was debugging this today and turned on catch on all exceptions. I was looking at some of them and looking into how the Retry mechanism works within the sdk. Because I was attached and artificially caused the request rate to temporarily slow down it recovered from the error and continued. I saw that there was some kind of metric that controls the delay when failures occur so I'm going to attempt to modify that.
I agree that it may be my internet, but as I said I've also seen similar intermittent failures when the lambda is deployed. If anything the fact that it causes it to repro semi-reliably is a good thing. The documentation on Retries and Timeouts is relatively sparse.
The Transfer Utility doesn't fit my needs since all of my data is in memory. All of the upload and download functions within it deal with using the disk.

@klaytaybai I see in DownloadCommand.async that there is a retry mechanism built on top of the underlying IAmazonS3.GetObjectAsync. Is it expected that clients should have to handle errors themselves and build that? The documentation and code doesn't make that clear but if that's the answer it's something I can do.

I spent today writing a mechanism for doing my own retries on top of S3 and it is now backing off requests and retrying failed requests correctly. To anyone who runs into a similar issue with S3, I recommend writing something on top of it that will retry your requests for you. Here was a basic rundown of what I did:

In the config set MaxErrorRetry to 0. This makes it so you don't have two separate systems trying to do retries.

Created something fairly generic to do retries:

	/// <summary>
	/// Interface for a task operation that can be retried on a delay.
	/// </summary>
	interface IRetriableTask<T>
	{
		/// <summary>
		/// Perform the operation.
		/// </summary>
		Task<T> Execute();

		/// <summary>
		/// Allows the class to decide whether the given exception can be retried. Return true to retry, false to throw
		/// it.
		/// </summary>
		bool ShouldRetryOnException(Exception in_exception);
	}

	class RetriableTask
	{
		/// <summary>
		/// Executes io_retriableTask multiple times using in_delaysBetweenRetries. If the task throws, this delays
		/// using the TimeSpan within in_delaysBetweenRetries that corresponds to the attempt number prior to trying
		/// again. Typically you want the delays to get progressively longer. Each failure will be logged to the
		/// Console. If we reach our max attempts, or io_retriableTask ShouldRetryOnException returns false the
		/// exception will be rethrown. Otherwise as soon as it succeeds the result is returned.
		/// </summary>
		public static async Task<T> ExecuteAsync<T>(IRetriableTask<T> io_retriableTask, List<TimeSpan> in_delaysBetweenRetries)
		{
			// Keep trying the operation until we either succeed or exceed retries.
			int attemptNum = 0;
			while (true)
			{
				// Attempt to Execute.
				System.Runtime.ExceptionServices.ExceptionDispatchInfo capturedException = null;
				try
				{
					return await io_retriableTask.Execute();
				}
				catch (Exception exception)
				{
					// This helps make callstacks easier to follow if this ends up rethrowing the exception.
					capturedException = System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(exception);
				}

				// We exceeded retries, or this exception should not be retried.
				if (attemptNum >= in_delaysBetweenRetries.Count ||
					!io_retriableTask.ShouldRetryOnException(capturedException.SourceException))
				{
					capturedException.Throw();
				}

				// Otherwise log the captured exception.
				Console.WriteLine($"{io_retriableTask.ToString()} failed attempt {attemptNum} with exception {capturedException.SourceException.ToString()}. Delaying and trying again.");

				// Wait the configured amount of time.
				await Task.Delay(in_delaysBetweenRetries[attemptNum]);

				// And retry.
				++attemptNum;
			}
		}
	}

		/// <summary>
		/// Base class for all of our S3 retriable tasks.
		/// </summary>
		private abstract class S3RetriableTask<T> : IRetriableTask<T>
		{
			protected S3RetriableTask(IAmazonS3 io_s3Client)
			{
				m_s3Client = io_s3Client;
			}

			public abstract Task<T> Execute();

			public bool ShouldRetryOnException(Exception in_exception)
			{
				// For now we retry on everything that isn't an AmazonS3Exception. Typically AmazonS3Exceptions are
				// reserved for unrecoverable errors such as missing keys. Retrying and swallowing these wouldn't be
				// good.
				if (in_exception is AmazonS3Exception)
				{
					return false;
				}

				return true;
			}

			protected IAmazonS3 m_s3Client;
		}

		/// <summary>
		/// A retriable task to retrieve an object from S3.
		/// </summary>
		private class GetObjectTask : S3RetriableTask<Amazon.S3.Model.GetObjectResponse>
		{
			public GetObjectTask(IAmazonS3 io_s3Client, string in_bucket, string in_key) :
				base(io_s3Client)
			{
				m_bucket = in_bucket;
				m_key = in_key;
			}

			public async override Task<Amazon.S3.Model.GetObjectResponse> Execute()
			{
				return await m_s3Client.GetObjectAsync(new Amazon.S3.Model.GetObjectRequest()
				{
					BucketName = m_bucket,
					Key = m_key
				});
			}

			private string m_bucket;
			private string m_key;
		}

It seems to be working pretty well except for a couple of kinks that I'm still working out. Thanks for the hints of where to look @klaytaybai.


Add custom header for config post call

Also fix the regex for stage triggered on time pattern

This is already covered by some is and template specs. Can we remove the corresponding functional tests?

I don't see much of value in functional tests asserting on a date/time format


NVM not working under Restricted network

NVM not working under Restricted network. How to set proxy for NVM?
$nvm ls-remote
NA

What do you mean "restricted network"?
Indeed if you can't connect to nodejs.org and iojs.org then you're going to have trouble. You can set the NVM_NODEJS_ORG_MIRROR and NVM_IOJS_ORG_MIRROR environment variables if you like, to point to mirrors or proxies that you are able to access.
These are documented near the bottom of the usage section in the readme.

I got it working by adding proxy

@byteshiva hello Siva, could you please kindly share what's need to be done in order to have it to work? I'm having the same issue (behind company proxy, nvm does not work). Thank you.

This is what is required
export http_proxy=username:password@proxy-server-your-company.com:port
export NVM_NODEJS_ORG_MIRROR=https://nodejs.org/dist

@byteshiva thank you so much Siva, turns out a https_proxy environment variable is required as well.
export https_proxy=username:password@proxy-server-your-company.com:port
It's all good now :)

Not enough.  Local company bad ssl certs.  How do I get nvm to do equivalent of (in .nmprc)
strict-ssl=false

nvm uses curl and ignores curlrc; you can’t get it to bypass SSL. You can use env variables to point to the http version of the mirrors (see the readme) but the better solution is to get your company to fix their network.

ALL_PROXY=http://proxyUrl:port nvm ls-remote
fixed it for me

This worked like a charm. Thanks @Cradanimal


No longer install hdbscan from git for docs

A new release containing the relevant fixes has been pushed.
Contributor Checklist:

[ ] I have added or updated my entry in the creators.json file
[ ] I have added a changelog entry for my contribution
[ ] I have added/updated documentation for all user-facing changes
[ ] I have added/updated test cases
[ ] I have included the rebuilt production build of the client (only if changes were made to the GUI)

Reviewer Checklist:

[ ] /azp run libertem.libertem-data passed

/azp run libertem.libertem-data


Remove unused log dependency

parking_lot is unused as well btw
i forgot to remove both when deprecating interpreter

Should I remove parking_lot  too?


[Mat-Select] Custom Font-family not get applied to mat-option

Bug, feature request, or proposal: For Mat-Select, custom Font-family not get applied to mat-option using panelClass property. You have to explicitly add style to mat-option. Even ngClass don't work for mat-option.
What is the expected behavior? If other stylings are getting applied using panelClass, font-family should also get applied
What is the current behavior? For Mat-Select, custom Font-family not get applied to mat-option using panelClass property. You have to explicitly add style to mat-option. Even ngClass don't work for mat-option.
What are the steps to reproduce?
https://stackblitz.com/edit/select-dropdown-panel-font-family-issue?file=app%2Fapp.component.html
What is the use-case or motivation for changing an existing behavior?
This will help in customizing the fonts for mat-select.
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
Angular - v5.x
Is there anything else we should know?

.mySelectedPanel .mat-option {
   font-family: 'Times New Roman';
   font-size: 15px;
}

See this StackBlitz for a working example.

Closing since it seems to be working as expected.

Thanks @donroyco for the solution.

.mySelectedPanel .mat-option {
   font-family: 'Times New Roman';
   font-size: 15px;
}

See this StackBlitz for a working example.

Well, changing the font-family works but a change of the font-size doesn't !?


⚠ ClamAV has degraded performance

In d7e8626, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 210 ms

Resolved: ClamAV performance has improved in 31c010a.


Parameterize github-hooks config

So parameterization may not work for trigger blocks in jjb.
Thoughts on doing a temporary side index (new yaml file) with the trial jobs? That way we can copy over the various blocks and hard-code it in the trigger definition.

@bstinsonmhk Couple changes..now it gets parameterized properly. Would appreciate one more review from you

:+1: let's try this


(Mobile+Web) Add Properties/add level should be Edit properties/edit level

because in Add properties you can delete properties and edit properties as well
similarly for Add levels

Update at commit cdbab23.


Size issue in HeaderMap::reserve()

https://github.com/hyperium/http/blob/9c05e391e00474abaa8c14a86bcb0fc5eff1120e/src/header/map.rs#L622-L640
usize::next_power_of_two() method silently overflows to 0 in release mode. This makes it possible to shrink the size of the map to 0 with HeaderMap::reserve().

If the map doesn't contain any entry, it sets the mask value to usize::MAX which is inconsistent but doesn't create any immediate harm.
If the map contains any entry, the code will call self.grow(0) and start infinite probing in this line.

Another problem is that the assertion for MAX_SIZE doesn't exist here, so it is possible to grow the map larger than MAX_SIZE.
Demonstration

@Qwaz Am I correct that the potential vulnerability this presents is that an attacker could cause a denial of service by sending a request which causes this overflow?

@hawkw An attacker can trigger a DoS if they can call HeaderMap::reserve() to non-empty HeaderMap with a controlled parameter. I believe this scenario is unlikely in practice, but yes, it's theoretically possible.


Missing DLL after Build

The readme mentions that this dll should work for a Windows Standalone build, but I have been unable to actually produce a working build of this project or my own which was was built with the existing BleApi.cs script. I am not terribly familiar with dll's in unity, and the available answers around the web are unclear to me. On each frame, I receive a DLLNotFoundException for BleWinrtDll.dll immediately at startup after building.  This is not an issue when using the Unity editor. Could anyone shed light on what I'm doing wrong?
To be clear, this is only after building the application and using it on a different device. The standalone will work on the device that is is built on.

Solved embarrassingly quickly after posting this. For anyone else that might have this problem. Please ensure that your application requires the latest Microsoft Visual C++ Redistributable to be installed.

Thank you eproasim 🙂, I added it to the readme.


pipenv sync fails without Pipfile present

Issue description
Running pipenv sync without a Pipfile present (the Pipfile.lock is there) fails with the message
Error: Pipfile.lock is missing! You need to run $ pipenv lock first.

Expected result
I would expect the Pipfile not to be needed at all for sync since it should just install the pinned versions from the Pipfile.lock
Actual result
pipenv won't install anything because it either thinks it needs a Pipfile as well as a Pipfile.lock for some reason. It also complains about the wrong file (Pipenv.lock).
Steps to replicate

Create a new virtualenv, e.g with pipenv install requests
Remove the Pipfile (keeping only the Pipfile.lock)
Run pipenv sync

$ pipenv --support
Pipenv version: '2018.7.1'
Pipenv location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/lib/python3.7/site-packages/pipenv'
Python location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python3.7'
Other Python installations in PATH:

2.7: /usr/local/bin/python2.7

2.7: /usr/local/bin/python2.7

2.7: /usr/bin/python2.7

2.7: /usr/local/bin/python2.7

2.7: /usr/bin/python2.7

3.7: /usr/local/bin/python3.7m

3.7: /usr/local/bin/python3.7

3.7: /usr/local/bin/python3.7

2.7.15: /usr/local/bin/python

2.7.10: /usr/bin/python

2.7.15: /usr/local/bin/python

2.7.10: /usr/bin/python

2.7.15: /usr/local/bin/python2

2.7.15: /usr/local/bin/python2

3.7.0: /usr/local/bin/python3

3.7.0: /usr/local/bin/python3

PEP 508 Information:
{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.5.0: Fri Apr 13 19:32:32 PDT '
                     '2018; root:xnu-4570.51.2~1/RELEASE_X86_64',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

PATH
TERM_PROGRAM
SHELL
TERM
TMPDIR
PIPENV_VENV_IN_PROJECT
Apple_PubSub_Socket_Render
TERM_PROGRAM_VERSION
TERM_SESSION_ID
LC_ALL
USER
SSH_AUTH_SOCK
__CF_USER_TEXT_ENCODING
LSCOLORS
_
PWD
LANG
ITERM_PROFILE
XPC_FLAGS
XPC_SERVICE_NAME
COLORFGBG
HOME
SHLVL
ITERM_SESSION_ID
LOGNAME
LC_CTYPE
GOPATH
DISPLAY
COLORTERM
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH

Pipenv–specific environment variables:

PIPENV_VENV_IN_PROJECT: 1

Debug–specific environment variables:

PATH: /usr/local/Cellar/pipenv/2018.7.1/libexec/tools:/Users/ulrjoh2/.composer/vendor/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS:/usr/local/opt/go/libexec/bin:/Users/ulrjoh2/work/bin:/Users/ulrjoh2/.local/bin:/Users/ulrjoh2/.composer/vendor/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS:/usr/local/opt/go/libexec/bin:/Users/ulrjoh2/work/bin
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /tmp/test

From a quick look at the source, it seems that the reason for this behaviour is that pipenv figures out where Pipfile.lock is by first searching for Pipfile. When it can't find a Pipfile, it gives up and throws the error message about the missing lock file.

Honestly I am not sure install --ignore-pipfile should do after #1463 is fully implemented. There really should be a full-on review on various install commands and options. But from my understanding the original purpose of --ignore-pipfile is actually to ignore the process of checking whether lock is up-to-date to Pipfile, not Pipfile entirely (and no, it’s not named very intuitively), so it shouldn’t accept a missing Pipfile. The best behaviour is probably to error without one.
Review of the install command is far more involved than fixing the sync command. I would recommend creating a PEEP to fully discuss it. This thread would focus on fixing the sync command. The fix is not very difficult IMO, so I’ll slap a good first issue tag on it. Contribution are welcomed!

I just found #1977 which seems to also mention this exact behaviour, see comment https://github.com/pypa/pipenv/issues/1977#issuecomment-402213517
Should this be closed as a duplicate of that?

I would say sync should stop looking for Pipfile.

I've tinkered a little with just copying the find() code from the patched pipfile module and using that for finding the lock-file in pipenv/project.py, and that seems to solve the problem of not finding the lockfile, but since do_sync() runs ensure_project(), an empty Pipfile is always created when running the sync command.
Getting rid of all the uses of Pipfile in all the functions called by do_sync() quickly grew to a much larger project, so I'm not sure that should be included in a fix for this issue. Perhaps that should go with solving #41 that you mentioned above.
Is it reasonable to post a PR just fixing the lockfile find problem? In that case I'll try and get one ready within the next day or two.

Hi - Is that still open? I'll be happy to assist with that if help is still required

This is open.

It looks like this issue is at least partly solved by https://github.com/pypa/pipenv/pull/3386,
I no longer get an error about a missing Pipfile or lockfile when I do a pipenv sync with only the Pipfile.lock present.
There is still weirdness remaining tough, like the pipenv sync creates a Pipfile just like when running pipenv install --ignore-pipfile , that I described in a comment further up: https://github.com/pypa/pipenv/issues/2834#issuecomment-419699150
So these two commands seem to behave the same way now.
The subject of this issue only mentions the sync failing, which is no longer the case. Should this issue be closed?

Sure thing, thank you for following up


Updated line 21

added or modify  to line 21 here:
When Azure Policy runs the template in the deployIfNotExists or modify policy definition, it does so using.
This applies to deployIfNotExists and modify policy definition as well.

@Buchatech : Thanks for your contribution! The author(s) have been notified to review your proposed change.


YUNIKORN-42 (Work in progress changes for recorder)

Just added a WIP change for recorder, which includes:

Recorder implementation.
How to use it:

diagnostic. GetEventsRecorder(). RecordResourceRequestEvent(...)

Changes in api package

Changes in RMProxy

Pending:

Haven't added any recording message to scheduler yet.
Haven't properly setup golint/IDE plugin, so the format could be wrong.

@wilfred-s , @TaoYang526 , @yangwwei  please add your suggestions.


Enable shutdown of Tracker threads (close #297)

For issue #297. A fix for issue #291 / PR #293.
The threads in the ScheduledThreadPool ExecutorService in the Tracker can now be stopped. This was causing the simple-console demo to hang.
The new method tracker.close() shuts down the Tracker threadpool, and also calls the BatchEmitter method emitter.close(), which interrupts the Emitter threads.

Thanks for your pull request. Is this your first contribution to a Snowplow open source project? Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
:memo: Please visit https://docs.snowplowanalytics.com/docs/contributing/contributor-license-agreement/ to learn more and sign.
Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.


chore(deps): run helm dep up for kube-monitoring and alerts

Can you update the PluginDefinitions with the new chart versions?


Update NetworkInfoProvider.kt

Fix for API 34

Closing this branch PR as there was already a fix implemented for this. Thanks @nstratak for opening the PR.


feat: add prefetchMethod option

Some services (such as AWS) provide URLs that works only for one pre-configured verb.
By default, react-doc-viewer fetches document metadata through a HEAD request in order to guess its Content-Type.
If you need to have a specific verb for the pre-fetching, use the prefetchMethod option on the DocViewer:
import DocViewer from "react-doc-viewer";

<DocViewer
  prefetchMethod="GET"
/>;

any update on this? this would be good

Hey @mattmogford-alcumus 👋
Any news on this front? Thanks 🙏

Any chance this PR gets merged some time soon? 😅

Please???

@crice88 @adrael @itsmitchyyy I forked this repository and added changes from this pull request. Additionally, I added the ability to overwrite the component with loading renderer and no renderer (error). This fork is here and has the ability to add from npm: https://github.com/cyntler/react-doc-viewer


Detailed Visual Studio instructions in Readme

To debug one of the executables in visual studio, several steps must be taken on a per-machine basis.

The executable to be run must be set as the "Startup Project" by right clicking on the project and selecting "Set as Start Up Project", otherwise the debugger will try to run the static library file...
Any command line arguments will need to be set on a per-machine basis in the debug settings menu. This must be done per configuration (debug/release/profile(future))

I'll fix this at some point, only question is whether we want to include screenshots in the readme.


Fix cipher context assignment in handlecipherconfig.go

The previous version was always assigning the last cipher context in the slice instead of the one with the correct ID.

Thanks.
Have you seen some failure this would cause, or this is purely from a code review? If the former we should try to understand where it shoyld be back-ported to lts releases.

I noticed the bug, while testing the change of signing certificate in the controller -> it introduces more cipher contexts and EVE ends up choosing the wrong one for the decryption.
Since the certificate change in the controller is independent from EVE, this can also affect the older EVE versions.


[DEVHAS-249] [DEVHAS-256] [DEVHAS-258] CDQ should return git link, Component should check for dup port, SEB preserves base container name for overlays resource

Signed-off-by: Maysun J Faisal<EMAIL_ADDRESS>What does this PR do?:

249
CDQ detection should return github links instead of devfile registry links for a detected sample.
The reason we do this is because if a devfile registry link is returned for instance https://registry.devfile.io/devfiles/python-basic, the outerloop Devfile Kubernetes file deploy.yaml is not available at https://registry.devfile.io/devfiles/deploy.yaml
But instead of returning a devfile registry link, we return the corresponding github raw link https://raw.githubusercontent.com/devfile-samples/devfile-sample-python-basic/main/devfile.yaml, then deploy.yaml is available at https://raw.githubusercontent.com/devfile-samples/devfile-sample-python-basic/main/deploy.yaml and a Component create should be fine.
256
Looped through the Service and Container ports to check if they were present, before appending
258
Pulls in the latest gitops generator changes updated by @yangcao77 at https://github.com/redhat-developer/gitops-generator/pull/39
Which issue(s)/story(ies) does this PR fixes:

https://issues.redhat.com/browse/DEVHAS-249
https://issues.redhat.com/browse/DEVHAS-256
https://issues.redhat.com/browse/DEVHAS-258
PR acceptance criteria:

[x] Unit/Functional tests

[ ] Documentation

[ ] Client Impact

How to test changes / Special notes to the reviewer:

Tests should pass
Updated tests to check for devfileUrl and dockerfileUrl (249)
Updated tests to check for duplicate ports (256)
Tests were updated on gitops gen library (258)
Update storageLimit & storageRequest tests to increase code coverage

/retest

Also pulls in gitops library changes by @yangcao77 from https://github.com/redhat-developer/gitops-generator/pull/39

/retest

/retest


Feature/map 236/add infinity scroll to comments

To use functional you should run yarn install on the frontEnd to install react-intersection-observer library.
Added separately download parent comments and downloading repleis on click
You can check it on "Білокузьминівські скелі" location, I added few comment here.
PS: on old comments it probably dont working  , because I add new field to Comments mode

When a user added a comment it moves down in the range, and after uploading a new portion of comments they stand in the right way  .

Fixed


add slack-generic-client executable

Two changes in this commit:

I added a stack.yaml so we can build with stackage.
I created an executable stack-generic-client that one can use to probe the Stack API using the client library. This is mainly for development purposes, but it might evolve into a more full-featured command line stack client.

Thanks! :+1: Looks good


Make binaryen an external dependency

Removed the binaryen submodule, added instructions to compile and install binaryen, and changed cmake to find binaryen binaries using BINARYEN_ROOT var or else find them on path.

Took 19 minutes.  Could knock off another 2 minutes if we prebuild boost.
https://travis-ci.org/EOSIO/eos/builds/270846270

#297


Lifei/fix GitHub workflow triggers

Why
Currently we don't have mechanism to create the version tag and publish the package for new version
What

Enabled build on main branch
Created a script file to check whether the version tag already exists
Create a version tag when main branch passes the build and no same version tag exists
When publishing the package, checkout the code with the tag version

Note
I heard we are also going to discuss the semantic version for auto version.  Before the decision on the semantic version, we may use this mechanism in the PR, at least we can release new versions

ok to be clear, goose tells me this is how it works:
                                                                Detailed Workflow Summary:

 1 Merge to main:
    • Upon merging a PR to the main branch, the push event triggers both workflows.
 2 CI Workflow Execution:
    • Build Job:
       • The code is checked out and built.
       • Tests run to ensure everything is working correctly.
    • Create Tag Job:
       • The code is checked out again.
       • Required dependencies and tools are installed.
       • The check_tag_exists.sh script runs:
          • Extracts version number from the distribution file.
          • Checks if the extracted version tag already exists in the repository.
       • Since the version hasn't changed, it detects that the version tag already exists.
       • Tag Creation:
          • As the version tag already exists, the tag creation step is bypassed (i.e., no new tag is created).
 3 PyPI Release Workflow Execution:
    • Checkout with Ref:
       • Checks out the code with the specific reference related to the merge.
    • Publish Step:
       • Installs dependencies.
       • Attempts to publish the package.
       • As there’s no new version, the package will not be republished to PyPI if it already exists.

                                                                        Conclusion

With every merge to main, the CI workflow ensures the code is built, tested, and checks for an existing version tag. If the version has not changed, it
skips creating a new tag. Concurrently, the PyPI release workflow checks out the code and attempts to publish it, but it respects pre-existing versions,
avoiding redundant uploads.

is that right? that seems ok then

                                                                Detailed Workflow Summary:

 1 Merge to main:
    • Upon merging a PR to the main branch, the push event triggers both workflows.

The PyPI Release Workflow Execution is only triggered when a tag name starts with v is pushed

3 PyPI Release Workflow Execution:
• Checkout with Ref:
• Checks out the code with the specific reference related to the merge.
It check out the code with the tag which was pushed

Concurrently, the PyPI release workflow checks out the code and attempts to publish it, but it respects pre-existing versions,
avoiding redundant uploads.
The Pypi release workflow does not run concurrently with the CI workflow.  It is only trigged by pushing a tag name starts with v

                                                                Detailed Workflow Summary:

 1 Merge to main:
    • Upon merging a PR to the main branch, the push event triggers both workflows.

The PyPI Release Workflow Execution is only triggered when a tag name starts with v is pushed

3 PyPI Release Workflow Execution:
• Checkout with Ref:
• Checks out the code with the specific reference related to the merge.

It checks out the code with the tag which was pushed

Concurrently, the PyPI release workflow checks out the code and attempts to publish it, but it respects pre-existing versions,
avoiding redundant uploads.

The Pypi release workflow does not run concurrently with the CI workflow. It is only trigged by pushing a tag name starts with v

This is no long valid now.   I've pushed a commit to move the create_tag job to the pypi_release workflow now

Nice, I think this makes sense. However, I think initially, we probably want to manually publish tags for releases so we don't push anything broken? Or I guess the expecation is you don't version bump everything? We def need to follow up with the semver release strategy.
Also, here's a similar PR https://github.com/square/goose/pull/19/files -- I have some checks in there for checking versions in the pyproject and on pypi which maybe we just move in here?

@zakiali yeah - unless there was a goose-unstable package for people to live on the edge, but probably overkill, as long as version number changes then it could be a release? woudl be good to have more releases (of goose + exchange) regularly ig possible

I think the semver/conventional commits stuff will help here greatly to have a standard practice. We can have release candidates pushed up for experimental/exploratory installs, while still having stable releases for users not wanting to be on the cutting edge. For now, releases on version bump are 👍

ok so this should have the publish only triggered on tag now? (and then check it builds of course)


rotatelogs: add an option to generate log file name with timestemp

In some cases, we prefer to create a log file with the file name generated using hours, minutes, and seconds.
The suffix .1, .2, .3, etc., is not needed in this case.
📑 Description
Log suffixes carrying time information can be used to organize and collect logs more conveniently.

@coderabbitai summary

✅ Checks

[x] Added test cases for my code

@coderabbitai review


fix: added useEffect cleanup in useRegisterBindings

Describe your changes
Issue related widget states become stale on component re-render
for example,
added a button on showDialog, opened showDialog and set that button loader to true, close the showDialog and open again, try again to set loader, and this time nothing happen due to setter function become stale.
Screenshots [Optional]
after:
https://github.com/user-attachments/assets/044425a9-82c9-4c46-a669-b7ea5d25d922
Issue ticket number and link
Closes #861
Checklist before requesting a review

[x] I have performed a self-review of my code
[ ] I have added tests
[x] I have added a changeset pnpm changeset add
[x] I have added example usage in the kitchen sink app

@sagardspeed2 this will create infinite renders and cause other dependent expressions to fail. See the failing test. Just update the invokables each time.


[SUPPORT] NotSerializableException using SparkRDDWriteClient with OCC and DynamoDBBasedLockProvider

Describe the problem you faced
We are trying to delete records from a MOR table with the following write configs for locking and concurrency and are running into java.io.NotSerializableException: org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
write concurrency/locking configs:
(hoodie.write.concurrency.mode,OPTIMISTIC_CONCURRENCY_CONTROL)
(hoodie.write.lock.conflict.resolution.strategy,org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy)
(hoodie.write.lock.dynamodb.endpoint_url,http://localhost:8000)
(hoodie.write.lock.dynamodb.partition_key,users_changes-us-east-1-local)
(hoodie.write.lock.dynamodb.region,us-east-1)
(hoodie.write.lock.dynamodb.table,datalake-locks)
(hoodie.write.lock.provider,org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider)

(Note the config above is taken from an integration test where we specify a local dynamoDB but the same error happens in production dynamoDB).
delete code snippet:
val writeClient: SparkRDDWriteClient[_] = buildWriteClient(config, spark.sparkContext)
    try {
      val deleteInstant = writeClient.startCommit()
      writeClient.delete(recordKeys, deleteInstant)
...

To Reproduce

Build an RDD of recordKeys to delete
Build a SparkRDDWriteClient with concurrency and locking configs as specified above
execute writeClient.delete(recordKeys, deleteInstant)
observe java.io.NotSerializableException: org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider

Expected behavior
delete is successful
Environment Description
Hudi version : 0.13.0
Spark version : 3.3.0
Hive version : n/a
Hadoop version : 3.2.1
Storage (HDFS/S3/GCS..) : s3
Running on Docker? (yes/no) : no
Additional context
We are using the writeclient to delete an RDD of recordKeys because the records keys were generated using UUID keys, which cannot be deleted using the dataframe API. This was surfaced in this issue. It seems we did not test with full concurrency/locking configured at that time.
Stacktrace
org.apache.hudi.exception.HoodieUpsertException: Failed to delete for commit time<PHONE_NUMBER>0655498
org.apache.hudi.exception.HoodieUpsertException: Failed to delete for commit time<PHONE_NUMBER>0655498
        at org.apache.hudi.table.action.commit.HoodieDeleteHelper.execute(HoodieDeleteHelper.java:117)
        at org.apache.hudi.table.action.deltacommit.SparkDeleteDeltaCommitActionExecutor.execute(SparkDeleteDeltaCommitActionExecutor.java:45)
        at org.apache.hudi.table.HoodieSparkMergeOnReadTable.delete(HoodieSparkMergeOnReadTable.java:105)
        at org.apache.hudi.table.HoodieSparkMergeOnReadTable.delete(HoodieSparkMergeOnReadTable.java:80)
        at org.apache.hudi.client.SparkRDDWriteClient.delete(SparkRDDWriteClient.java:243)
        at com.heap.datalake.delete.UUIDRecordKeyDeleter$.deleteRecords(UUIDRecordKeyDeleter.scala:82)
        at com.heap.datalake.delete.UUIDRecordKeyDeleter$.deleteRecords(UUIDRecordKeyDeleter.scala:210)
        at com.heap.datalake.delete.UserPropertiesDeleter$.deleteUsers(UserPropertiesDeleter.scala:48)
        at com.heap.datalake.delete.UserPropertiesDeleter$.run(UserPropertiesDeleter.scala:23)
        at com.heap.datalake.delete.DeleteUsersApp$.$anonfun$run$2(DeleteUsersApp.scala:56)
        at scala.util.Try$.apply(Try.scala:213)
        at com.heap.datalake.delete.DeleteUsersApp$.run(DeleteUsersApp.scala:56)
        at com.heap.datalake.delete.DeleteUsersApp$.$anonfun$main$2(DeleteUsersApp.scala:38)
        at com.heap.datalake.delete.DeleteUsersApp$.$anonfun$main$2$adapted(DeleteUsersApp.scala:23)
        at scala.util.Success.foreach(Try.scala:253)
        at com.heap.datalake.delete.DeleteUsersApp$.main(DeleteUsersApp.scala:23)
        at com.heap.datalake.delete.DeleteUsersApp.main(DeleteUsersApp.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742)
Caused by: org.apache.spark.SparkException: Task not serializable
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:444)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:416)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:163)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:2493)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2268)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2294)
        at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1021)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:1020)
        at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362)
        at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361)
        at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
        at org.apache.hudi.data.HoodieJavaRDD.collectAsList(HoodieJavaRDD.java:163)
        at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.setCommitMetadata(BaseSparkCommitActionExecutor.java:283)
        at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.autoCommit(BaseCommitActionExecutor.java:186)
        at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.commitOnAutoCommit(BaseCommitActionExecutor.java:174)
        at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.updateIndexAndCommitIfNeeded(BaseSparkCommitActionExecutor.java:273)
        at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:178)
        at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.execute(BaseSparkCommitActionExecutor.java:83)
        at org.apache.hudi.table.action.commit.HoodieDeleteHelper.execute(HoodieDeleteHelper.java:103)
        ... 21 more
Caused by: java.io.NotSerializableException: org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
Serialization stack:
        - object not serializable (class: org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider, value: org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider@697af25)
        - field (class: org.apache.hudi.client.transaction.lock.LockManager, name: lockProvider, type: interface org.apache.hudi.common.lock.LockProvider)
        - object (class org.apache.hudi.client.transaction.lock.LockManager, org.apache.hudi.client.transaction.lock.LockManager@a109d65)
        - field (class: org.apache.hudi.client.transaction.TransactionManager, name: lockManager, type: class org.apache.hudi.client.transaction.lock.LockManager)
        - object (class org.apache.hudi.client.transaction.TransactionManager, org.apache.hudi.client.transaction.TransactionManager@157ce6e9)
        - field (class: org.apache.hudi.table.action.commit.BaseCommitActionExecutor, name: txnManager, type: class org.apache.hudi.client.transaction.TransactionManager)
        - object (class org.apache.hudi.table.action.deltacommit.SparkDeleteDeltaCommitActionExecutor, org.apache.hudi.table.action.deltacommit.SparkDeleteDeltaCommitActionExecutor@d63f085)
        - element of array (index: 0)
        - array (class [Ljava.lang.Object;, size 2)
        - field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
        - object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=class org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor, functionalInterfaceMethod=org/apache/spark/api/java/functi
on/Function2.call:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;, implementation=invokeSpecial org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1:(Lorg/apache/sp
ark/Partitioner;Ljava/lang/Integer;Ljava/util/Iterator;)Ljava/util/Iterator;, instantiatedMethodType=(Ljava/lang/Integer;Ljava/util/Iterator;)Ljava/util/Iterator;, numCaptured=2])
        - writeReplace data (class: java.lang.invoke.SerializedLambda)
        - object (class org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor$$Lambda$5283/1488307125, org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor$$Lambda$5283/1488307125@467af2d8)
        - element of array (index: 0)
        - array (class [Ljava.lang.Object;, size 1)
        - field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
        - object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=interface org.apache.spark.api.java.JavaRDDLike, functionalInterfaceMethod=scala/Function2.apply:(Ljava/lang/Object;Ljava/lang/Obje
ct;)Ljava/lang/Object;, implementation=invokeStatic org/apache/spark/api/java/JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted:(Lorg/apache/spark/api/java/function/Function2;Ljava/lang/Object;Lscala/collection/Iterato
r;)Lscala/collection/Iterator;, instantiatedMethodType=(Ljava/lang/Object;Lscala/collection/Iterator;)Lscala/collection/Iterator;, numCaptured=1])
        - writeReplace data (class: java.lang.invoke.SerializedLambda)
        - object (class org.apache.spark.api.java.JavaRDDLike$$Lambda$5284/280417748, org.apache.spark.api.java.JavaRDDLike$$Lambda$5284/280417748@1c7adacb)
        - element of array (index: 0)
        - array (class [Ljava.lang.Object;, size 1)
        - field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
        - object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=class org.apache.spark.rdd.RDD, functionalInterfaceMethod=scala/Function3.apply:(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Obj
ect;)Ljava/lang/Object;, implementation=invokeStatic org/apache/spark/rdd/RDD.$anonfun$mapPartitionsWithIndex$2$adapted:(Lscala/Function2;Lorg/apache/spark/TaskContext;Ljava/lang/Object;Lscala/collection/Iterator;)Lscala/c
ollection/Iterator;, instantiatedMethodType=(Lorg/apache/spark/TaskContext;Ljava/lang/Object;Lscala/collection/Iterator;)Lscala/collection/Iterator;, numCaptured=1])
        - writeReplace data (class: java.lang.invoke.SerializedLambda)
        - object (class org.apache.spark.rdd.RDD$$Lambda$4810/1052938329, org.apache.spark.rdd.RDD$$Lambda$4810/1052938329@3c31aa92)
        - field (class: org.apache.spark.rdd.MapPartitionsRDD, name: f, type: interface scala.Function3)
        - object (class org.apache.spark.rdd.MapPartitionsRDD, MapPartitionsRDD[146] at mapPartitionsWithIndex at BaseSparkCommitActionExecutor.java:249)
        - field (class: org.apache.spark.NarrowDependency, name: _rdd, type: class org.apache.spark.rdd.RDD)
        - object (class org.apache.spark.OneToOneDependency, org.apache.spark.OneToOneDependency@373e82e8)
        -         - writeObject data (class: scala.collection.immutable.List$SerializationProxy)
        - object (class scala.collection.immutable.List$SerializationProxy, scala.collection.immutable.List$SerializationProxy@44652f9f)
        - writeReplace data (class: scala.collection.immutable.List$SerializationProxy)
        - object (class scala.collection.immutable.$colon$colon, List(org.apache.spark.OneToOneDependency@373e82e8))
        - field (class: org.apache.spark.rdd.RDD, name: dependencies_, type: interface scala.collection.Seq)
        - object (class org.apache.spark.rdd.MapPartitionsRDD, MapPartitionsRDD[147] at flatMap at BaseSparkCommitActionExecutor.java:255)
        - field (class: org.apache.spark.NarrowDependency, name: _rdd, type: class org.apache.spark.rdd.RDD)
        - object (class org.apache.spark.OneToOneDependency, org.apache.spark.OneToOneDependency@66f562d4)
        - writeObject data (class: scala.collection.immutable.List$SerializationProxy)
        - object (class scala.collection.immutable.List$SerializationProxy, scala.collection.immutable.List$SerializationProxy@d993b51)
        - writeReplace data (class: scala.collection.immutable.List$SerializationProxy)
        - object (class scala.collection.immutable.$colon$colon, List(org.apache.spark.OneToOneDependency@66f562d4))
        - field (class: org.apache.spark.rdd.RDD, name: dependencies_, type: interface scala.collection.Seq)
        - object (class org.apache.spark.rdd.MapPartitionsRDD, MapPartitionsRDD[148] at map at HoodieJavaRDD.java:111)
        - element of array (index: 0)
        - array (class [Ljava.lang.Object;, size 1)
        - field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
        - object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=class org.apache.spark.rdd.RDD, functionalInterfaceMethod=scala/Function1.apply:(Ljava/lang/Object;)Ljava/lang/Object;, implementat
ion=invokeStatic org/apache/spark/rdd/RDD.$anonfun$collect$2:(Lorg/apache/spark/rdd/RDD;Lscala/collection/Iterator;)Ljava/lang/Object;, instantiatedMethodType=(Lscala/collection/Iterator;)Ljava/lang/Object;, numCaptured=1]
)
        - writeReplace data (class: java.lang.invoke.SerializedLambda)
        - object (class org.apache.spark.rdd.RDD$$Lambda$3667/1073189054, org.apache.spark.rdd.RDD$$Lambda$3667/1073189054@770c17a4)
        at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:41)
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:49)
        at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:115)
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:441)
        ... 42 more
23/09/27 23:07:54 INFO SparkContext: Invoking stop() from shutdown hook

@ehurheap Did you tried the same lock configuration with a normal insert on test table to ensure configurations are good?

Yes. I can write a dataframe to the same table, for example:
    data.write
      .format("org.apache.hudi.Spark32PlusDefaultSource")
      .options(writeWithLocking)
      .mode("append")
      .save(tablePath)

where writeWithLocking options are:
(hoodie.bulkinsert.shuffle.parallelism,2)
(hoodie.bulkinsert.sort.mode,NONE)
(hoodie.clean.async,false)
(hoodie.clean.automatic,false)
(hoodie.cleaner.policy.failed.writes,LAZY)
(hoodie.combine.before.insert,false)
(hoodie.compact.inline,false)
(hoodie.compact.schedule.inline,false)
(hoodie.datasource.compaction.async.enable,false)
(hoodie.datasource.write.hive_style_partitioning,true)
(hoodie.datasource.write.keygenerator.class,org.apache.spark.sql.hudi.command.UuidKeyGenerator)
(hoodie.datasource.write.operation,bulk_insert)
(hoodie.datasource.write.partitionpath.field,env_id,week)
(hoodie.datasource.write.precombine.field,schematized_at)
(hoodie.datasource.write.recordkey.field,env_id,user_id)
(hoodie.datasource.write.row.writer.enable,false)
(hoodie.datasource.write.table.type,MERGE_ON_READ)
(hoodie.metadata.enable,false)
(hoodie.table.name,users_changes)
(hoodie.write.concurrency.mode,OPTIMISTIC_CONCURRENCY_CONTROL)
(hoodie.write.lock.dynamodb.endpoint_url,http://localhost:8000)
(hoodie.write.lock.dynamodb.partition_key,users_changes-us-east-1-local)
(hoodie.write.lock.dynamodb.region,us-east-1)
(hoodie.write.lock.dynamodb.table,datalake-locks)
(hoodie.write.lock.provider,org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider)

These locking configs are also in our production ingestion which writes to hudi using spark structured streaming without error.

@ehurheap Sorry for the delay here. Is it possible to provide full implementation of UUIDRecordKeyDeleter.scala

object UUIDRecordKeyDeleter {
  private val log = loggerForClass(UUIDRecordKeyDeleter.getClass)

  def query(tablePath: String, predicate: Column, queryType: String)(implicit
    spark: SparkSession
  ): DataFrame = {
    spark.read
      .format("hudi")
      .option(QUERY_TYPE.key(), queryType)
      .option(HoodieMetadataConfig.ENABLE.key(), "false")
      .load(tablePath)
      .where(predicate)
  }

  // Computes record keys of the records matching the predicate
  def computeRecordKeys(
    tablePath: String,
    predicate: Column
  )(implicit
    spark: SparkSession
  ): JavaRDD[HoodieKey] = {
    implicit val hoodieKeyEncoder: Encoder[HoodieKey] =
      Encoders.bean(classOf[org.apache.hudi.common.model.HoodieKey])
    query(tablePath, predicate, QUERY_TYPE_SNAPSHOT_OPT_VAL)
      .select(HoodieRecord.RECORD_KEY_METADATA_FIELD, HoodieRecord.PARTITION_PATH_METADATA_FIELD)
      .map { row =>
        {
          val recordKey = row.getString(0)
          val partitionPath = row.getString(1)
          new org.apache.hudi.common.model.HoodieKey(recordKey, partitionPath)
        }
      }
      .toJavaRDD
  }

  // Deletes all records with the provided record keys without triggering compaction.
  def deleteRecords(
    config: WriteClientConfig,
    envConfig: EnvConfig,
    dataLakeRecord: DataLakeRecord[_, _],
    recordKeys: JavaRDD[HoodieKey]
  )(implicit
    spark: SparkSession
  ): DeleteStats = {
    var stats = DeleteStats()
    val writeClient =
      buildWriteClient(config, envConfig, dataLakeRecord, spark.sparkContext)
    var deleteInstant: String = ""

    try {
      deleteInstant = writeClient.startCommit()
      val statuses: mutable.Seq[WriteStatus] =
        writeClient.delete(recordKeys, deleteInstant).collect().asScala
      stats = stats.copy(
        totalDeleted = getTotalDeletesFromWriteStatuses(statuses),
        totalPartitionsDeleted = getPartitionsFromWriteStatuses(statuses).size
      )
    } catch {
      case t: Throwable =>
        logErrorAndExit(s"Delete operation failed for instant $deleteInstant due to ", t)
    } finally {
      log.info(s"Finished delete operation for instant $deleteInstant")
      writeClient.close()
    }
    stats
  }

  def compactReadOptimizedRecordsMatchingPredicate(
    config: WriteClientConfig,
    envConfig: EnvConfig,
    dataLakeRecord: DataLakeRecord[_, _],
    predicate: Column
  )(implicit spark: SparkSession): DeleteStats = {
    import spark.implicits._
    var stats = DeleteStats()
    if (config.scheduleCompaction) {
      val partitions = query(config.tablePath, predicate, QUERY_TYPE_READ_OPTIMIZED_OPT_VAL)
        .select(col(HoodieRecord.PARTITION_PATH_METADATA_FIELD))
        .distinct()
        .map(r => r.getString(0))
        .collect()
        .toList

      if (partitions.isEmpty) {
        log.warn(
          s"Skipping compaction on table ${dataLakeRecord.tableName} as no records were found"
        )
      } else {
        log.info(
          s"Compacting ${partitions.size} partition paths in table ${dataLakeRecord.tableName}: ${partitions.take(10).mkString(",")}..."
        )

        val writeClient =
          compactingWriteClient(config, dataLakeRecord.tableName, envConfig, partitions)

        // Retry any pending (requested/inflight) compactions targeting overlapping partitions.
        // This is essentially a recovery process - if a previous run failed after scheduling a compaction,
        // the file groups in that compaction plan will be excluded from the new scheduled plan and their compaction won't be retried.
        // The compaction execution uses locks therefore it is safe to execute the same compaction plan concurrently.
        // The lake API limits GDPR deletion jobs to one at a time per env_id, therefore such conflict is unlikely (but still possible with mutations).
        // todo: get stats from this compaction to aggregate into final stats
        if (config.executeCompaction) {
          CompactionHelper.executeCompactionsWithOverlappingPartitions(
            config.tablePath,
            partitions,
            writeClient
          )
        }

        try {
          log.info(s"""Scheduling compaction on table ${dataLakeRecord.tableName}:
               |                 hoodie.compaction.strategy: ${writeClient.getConfig.getCompactionStrategy}
               |     "hoodie.compaction.include.partitions": ${writeClient.getConfig
                       .getString(
                         "hoodie.compaction.include.partitions"
                       )
                       .take(100)}...
               |             hoodie.compact.schedule.inline: ${writeClient.getConfig
                       .scheduleInlineCompaction()}
               |                      hoodie.compact.inline: ${writeClient.getConfig
                       .inlineCompactionEnabled()}
               |                 hoodie.write.lock.provider: ${writeClient.getConfig.getLockProviderClass}
               |                     hoodie.metadata.enable: ${writeClient.getConfig.getMetadataConfig
                       .enabled()}
               |""".stripMargin)
          val instant = writeClient.scheduleCompaction(org.apache.hudi.common.util.Option.empty())
          if (instant.isPresent) {
            if (config.executeCompaction) {
              log.info(
                s"Running compaction on table ${dataLakeRecord.tableName} at instant: ${instant.get}"
              )
              val metadata = writeClient.compact(instant.get).getCommitMetadata
              stats = stats.copy(
                compactionDeletes = metadata.get.getTotalRecordsDeleted,
                compactionTotalFiles = metadata.get.getTotalLogFilesCompacted,
                compactionTotalPartitions = metadata.get.fetchTotalPartitionsWritten
              )
              log.info(
                s"Successfully completed compaction on table ${dataLakeRecord.tableName} at instant: ${instant.get}"
              )
            }
          } else {
            log.warn(
              s"Unable to schedule compaction on table ${dataLakeRecord.tableName}, see Hudi logs for reason."
            )
          }
        } catch {
          case e: Throwable =>
            logErrorAndExit(s"Compaction failed on table ${dataLakeRecord.tableName}!", e)
        } finally {
          writeClient.close()
        }
      }
    }
    stats
  }

  // Deletes all records matching the predicate and runs compaction afterwards.
  // Deletion and compaction are only executed if there are records matching the predicate.
  def deleteRecords(
    config: WriteClientConfig,
    envConfig: EnvConfig,
    dataLakeRecord: DataLakeRecord[_, _],
    predicate: Column
  )(implicit
    spark: SparkSession
  ): DeleteStats = {
    var stats = DeleteStats()
    val recordCount = query(config.tablePath, predicate, QUERY_TYPE_SNAPSHOT_OPT_VAL).count()
    stats = stats.copy(
      totalFoundBeforeDelete = recordCount
    )
    if (recordCount == 0) {
      log.warn(
        s"Skipping delete operation on table ${dataLakeRecord.tableName} as no records were found"
      )
      DeleteStats()
    } else {
      log.warn(s"Deleting $recordCount records from table ${dataLakeRecord.tableName}")
      val keysToDelete = computeRecordKeys(config.tablePath, predicate)
      val deleteStats = deleteRecords(config, envConfig, dataLakeRecord, keysToDelete)
      stats = stats.copy(
        totalDeleted = deleteStats.totalDeleted,
        totalPartitionsDeleted = deleteStats.totalPartitionsDeleted
      )
    }
    val compactionStats =
      compactReadOptimizedRecordsMatchingPredicate(config, envConfig, dataLakeRecord, predicate)
    stats = stats.copy(
      compactionDeletes = compactionStats.compactionDeletes,
      compactionTotalFiles = compactionStats.compactionTotalFiles,
      compactionTotalPartitions = compactionStats.compactionTotalPartitions
    )
    stats
  }

  def buildWriteClient(
    config: WriteClientConfig,
    envConfig: EnvConfig,
    datalakeRecord: DataLakeRecord[_, _],
    sparkContext: SparkContext
  ): SparkRDDWriteClient[_] = {
    val lockProperties = new Properties()
    val lockOptionsMap =
      WriterOptions.lockOptions(datalakeRecord.tableName, config.region, envConfig: EnvConfig)
    lockProperties.putAll(lockOptionsMap.asJava)

    val metricsProperties = new Properties()
    val metricsOptionsMap = metricsOptions(config.statsDHost, envConfig, datalakeRecord.tableName)
    metricsProperties.putAll(metricsOptionsMap.asJava)

    val writerConfig = HoodieWriteConfig
      .newBuilder()
      .withCompactionConfig(
        HoodieCompactionConfig
          .newBuilder()
          .withInlineCompaction(false)
          .withScheduleInlineCompaction(false)
          .withMaxNumDeltaCommitsBeforeCompaction(1)
          .build()
      )
      .withArchivalConfig(HoodieArchivalConfig.newBuilder().withAutoArchive(false).build())
      .withCleanConfig(HoodieCleanConfig.newBuilder().withAutoClean(false).build())
      .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(false).build())
      .withLockConfig(HoodieLockConfig.newBuilder().fromProperties(lockProperties).build())
      .withMetricsConfig(HoodieMetricsConfig.newBuilder().fromProperties(metricsProperties).build())
      .withDeleteParallelism(config.writeParallelism)
      .withPath(config.tablePath)
      .forTable(datalakeRecord.tableName)
      .build()
    val engineContext: HoodieEngineContext = new HoodieSparkEngineContext(
      JavaSparkContext.fromSparkContext(sparkContext)
    )
    new SparkRDDWriteClient(engineContext, writerConfig)
  }

}

@ehurheap I was trying to reproduce and identify the reason for this Exception. Couldn't reproduce as was not able to run and even not able to identify exact reason. It is happening due to this custom code only.
This issue is occurring because the DynamoDBBasedLockProvider object is being utilized somewhere within an executor. When we directly pass the object, the problem arises. You might want to consider creating the object exclusively within the executor code if you can figure that out.

@ad1happy2go - thanks for the update. We will take another look.
I have an integration test that reproduces this but I'll need to extract from our codebase and repackage it so it can run standalone. I'll update here when I have that in case it helps.

Hi @ad1happy2go DynamoDBBasedLockProvider  and HiveMetastoreBasedLockProvider have the same issue like https://issues.apache.org/jira/browse/HUDI-3638, task not serializable in clean action.

@ad1happy2go here is a small repo with code and instructions on how to reproduce this problem. It would be great if you could try it out and let me know if that reproduced it for you.

Sure. I will give it a try.
On Sat, Dec 23, 2023 at 5:28 AM Liz Hurley @.***> wrote:

@ad1happy2go https://github.com/ad1happy2go here is a small repo
https://github.com/ehurheap/hudisupport with code and instructions on
how to reproduce this problem. It would be great if you could try it out
and let me know if that reproduced it for you.
—
Reply to this email directly, view it on GitHub
https://github.com/apache/hudi/issues/9807#issuecomment-1868139214, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/APD55YUDRLKABLYSTC5P4XTYKYNCHAVCNFSM6AAAAAA5MZ7V26VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGEZTSMRRGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


label is not optional

https://github.com/terra-money/terra.js/blob/d8711a3eb02d83d797a29fb94d361ca674108cff/src/core/wasm/msgs/MsgInstantiateContract.ts#L30
If instantiate without label, it will return this error
data: {
      code: 3,
      message: 'label is required: invalid request: invalid request',
      details: []
}

Oh it is because of classic sorry


Fix multires image scale when projection is WGS84.

Refs #566

@kimmat Can you confirm this branch resolves the issue for you?

I can confirm that this solved my issue. I have only tried this branch https://github.com/malban/mapviz/tree/fix-multires-image-scale-kinetic, as I use ROS Kinetic.
Thank you for solving the issue.


Best practice way to handle and redirect 50X and 404 errors when querying sanity fetch

Hey Nuxt Sanity team! I am looking for advice on error handling: When I query a page entry using the slug, I will need a way to do the following:

Redirect to the nuxt 404 error page if the query returns empty
Redirect and render the appropriate error when the query returns a 50X error

I am new to this module and I am not sure how best to do this. Can anyone share their method for doing so?
I am using Nuxt 3.0.0-rc.11 and Nuxt/sanity ^1.3.0

You probably want to do something like this: https://v3.nuxtjs.org/api/utils/create-error#throwing-errors-in-your-vue-app. There is also an error property returned by useSanityQuery which will contain the error (if you are using it on the server, or if the error has occurred on the client).

I'm closing this issue as I think I've answered it, but do let me know if I can be of any more help.

Hi @danielroe  apologies for the late response – been a busy few weeks. Thank you so much for this. I haven't had a chance to give it a go, but it certainly makes sense and I'm sure it answers my problem.

@danielroe  just looking at this now, but still having some difficulties (apologies for my ignorance here) – the sanity query docs are pretty light – I understand its a wrapper for AsyncData so tried to get my head around it from the nuxt async data docs but still a little unsure what I am doing wrong. From what I gather, the following should be about right?
const { pending, data:entry, error } = await useSanityQuery(query);
  console.log(pending); // returns null - should return true and then false
  console.log(error); // returns null - should return an object containing status
  if (error?.status !== '200') {
    throw createError({statusCode:error.status})
  } else if (!entry) {
    throw createError({statusCode:404})
  }

However, it's not working as I'd expect:

I'd expect pending to return true or false but it returns null.
I'd expect the error object to return {status:200} but it also returns null.

Can you tell me where I am going wrong?

Those are each refs, so you wouldn't directly check pending, error.status or entry - instead, you'd check error.value, pending.value and entry.value.
I can't reproduce the issue with your code (updated below and running in this module's playground) but if you provide a reproduction I'll happily take a look:
const { data: movies, pending, error } = await useAsyncData<QueryResult[]>('movies', () => sanity.fetch(query))
console.log(({ pending: pending.value, error: error.value, movies: movies.value }))

That worked @danielroe thank you. One last question though – the error doesn't provide an error status code – it's just a boolean. How do determine the status code? It would be good to display a different error message based on whether it's a network error as opposed to a server error?

You can follow https://github.com/nuxt/framework/pull/8521 - when this is merged, you can....


partially sort-of working titiler app using planetcantile

for local testing and development mostly, hope for a good demo out of this soon...

Thanks Vincent, that’s high praise, was actually wanting to contribute one or two parts back as a change to titiler. It’s really not all that different than what you have but for changing the parameter(s) to floats. The quantizer bit would be left out of course as it would mostly be up to the client to make that decision.- Dr. Andrew AnnexOn Feb 13, 2023, at 1:06 PM, Vincent Sarago @.***> wrote:
@vincentsarago commented on this pull request.
In planetcantile/topoalgo.py:

   z /= quantizer["resolution"]

   d_z = z / 256

   dd_z = z // 256

   ddd_z = dd_z // 256

   b = (d_z - dd_z) * 256

   g = ((dd_z / 256) - ddd_z) * 256

   r = ((ddd_z / 256) - (ddd_z // 256)) * 256

   arr = stack([r, g, b]).astype(uint8)

   return ImageData(

       arr,

       img.mask,

       assets=img.assets,

       crs=img.crs,

       bounds=img.bounds,

   )

That's so cool 😍
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

note, with https://github.com/developmentseed/titiler/pull/587 being merged the algorithm I added in this pr could be removed/simplified, I think. but merged it for now


Is this project unmaintained?

I'm under the impression, that the project is no longer maintained. There are quite a few open pull requests and Python3 support seems to be missing. I've also noticed some forks with changes and additions that could be handy.
Are there any plans for the future? Thanks!

We'd be willing to either help maintain this project or take over the pypi name.  Our fork has added test coverage, a few bug fixes, pth file support, and python 3 support.

Yelp has published its fork (which has the patches @asottile mentioned above) as https://pypi.python.org/pypi/virtualenv-tools3. We'd still be happy to help maintain this project or take over the PyPI name if anyone at @fireteam is interested :)
https://github.com/Yelp/virtualenv-tools
https://pypi.python.org/pypi/virtualenv-tools3

Cleaning up. If you think this should be re-opened, feel free to do so or contact me.


Tracker Data - Wrong SessionState

I am having trouble with the tracker example and the reason is that the application never goes into the correct xr.SessionState
When looping over rendered frames, you check the following:
if context.session_state == xr.SessionState.FOCUSED:
However, when I check the sessionState each frame, it is actually:
xr.SessionState.SYNCHRONIZED
Therefore, the example loops over the frames but never enters the if-statement
I looked at the session lifecycle but could not find a way to get it into the correct state:
https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#session-lifecycle
Any help would be greatly appreciated.

I believe examples using trackers require the use of OpenXR Actions. Actions can only be used in the FOCUSED state. My HTC Vive headset won't enter the FOCUSED state unless it thinks I am actively wearing the headset.  Maybe the same trick would work for you.
An alternative is the use the HEADLESS extension. I've been meaning to publish some headless examples but my focus is on other things at the moment.

Great thanks for the fast reply. That clears things up :)


"Escaping Literal Quotes in Strings" not working in IE11

Challenge Name

Escaping Literal Quotes in Strings
Issue Description

I've read through the bug reports to ensure I was writing the code correctly, and for some reason I can't get past this challenge.
Browser Information

Browser Name, Version: Internet Explorer 11
Operating System: Windows 7 Pro (64bit)
Mobile, Desktop, or Tablet: Desktop

Your Code
var myStr = "I am a \"double quoted\" string inside \"double quotes\"."; // Change this line

Just tried the same code in Google Chrome v51.0.2704.103 and it worked fine.  Issue is with Internet Explorer.

@9littlebees I can confirm your code works on Chrome. However, I don't have a Windows computer at the moment to reproduce your error.

It works for me on IE 11...
@9littlebees what version of IE are you using?

@heldersepu, he is using Internet Explorer 11, it says in the issue description :wink:

@erictleung Yep, first comment is my own, saying it works in Chrome.
@heldersepu No matter what I tried, I couldn't get it to work in IE 11, and I'm fully up-to-date with it.  Does your code match mine exactly?

I should have been more specific...
I'm on IE 11.0.9600.18349

@9littlebees yes my code is just a copy/paste of yours, take a look at my screenshot.

@heldersepu I'm on 11.0.9600.18314.  I'll update and try again.  I know the code is fine, because it worked flawlessly in Chrome.  I also don't see why a slightly older version of IE 11 (looks like it released 16-May-2016) should have problems with basic JavaScript, though...

You might want to change the Title to reflect that the problem is only on IE.
Are you familiar with the Developer Tools?
On IE you can press F12 and it will open, See if you are getting any errors there.
Also on the Developer Tools Console paste your string:
"I am a \"double quoted\" string inside \"double quotes\".";
it should respond with:
"I am a "double quoted" string inside "double quotes"."

Facing same problem with this challange but I am using crome.

@AmeySingh you need a space after inside.

I did that. Still not working.

@erictleung

I am having the exact same problem with Chrome on a mac.
I have the correct declaration that they want:
var myStr = "I am a "double quoted" string inside "double quotes".";
but I can't pass.  Could someone please help? I'm not sure how to get past this one.
And yes, I know that this is not the place for code help! I think this is a bug.
It does not pass the first requirement of two double quotes and four escaped quotes even though it does have all the required elements.


Bundle version served by dev-server to index.${platform}.js

Fixes #228 pending fix of https://github.com/facebook/react-native/issues/15791 (perhaps by way of merge of https://github.com/facebook/react-native/pull/15792 )
Going to want to give it a spin when moving parts are settled / write tests, but this is a general approach that should work.

and by index.${platform}.js in the commit message I mean index.${platform}.bundle.js

Instead of rewriting the URL, can't we set the content type?

Is there a way to set it such that it does not get smashed by webpack-dev-middleware setting it? If so, it would certainly be a preferable solution

Apparently webpack-dev-middleware has such a flag that I missed yesterday: https://github.com/webpack/webpack-dev-middleware/pull/216#issuecomment-327147606

Closing b/c #230 is a better solution.


Termdebug remote mode switch

To add a switch to allow Termdebug to send continue instead of run when doing :Termdebug!.
Use case:  when need to connect to a remote gdb server automatically though command line to launch Termdebug, it is possible to start Termdebug and manually do target remote ..., but it is annoying to do that especially if it needs to be repeated for several times.
  This does not work with use_prompt: 1 though.
g:gdb_x_opt, g:gdb_x_opt_value can be used to load extra file containing gdb command such as set up breakpoints etc.
It would be much nicer if all can be done with :Termdebug <host>:<port>, but that probbaly requries much more changes and it is over my current capability.

I may created the pull requests including a merge commit by mistake, don't know how to remove that, sorry.

I may created the pull requests including a merge commit by mistake,
don't know how to remove that, sorry.

Don't worry, individual commits rarely matter, I usually just look at
the final result.
--
hundred-and-one symptoms of being an internet addict:
192.
/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

To add a switch to allow Termdebug to send continue instead of run
when doing :Termdebug!.

This sounds like a useful addition.  I do have a few remarks.
Please also update the help.  Otherwise it will be hard to understand
how this works.  Or that the support even exists.  The examples you have
given here are a good start.
There is no need for "g:termdebug_remotemode".  The other
"g:termdebug_..." variables exist for backwards compatibility before
g:termdebug_config was added.  All new config options should go into
g:termdebug_config to avoid polluting the global namespace.
For using "=" in an assignment, please surround it with spaces, so that
it is easier to spot the separation between LHS and RHS.
Can "continue" be send with s:SendResumingCommand() ?  Then s:stopped
will be updated properly.  And it will work with a prompt buffer.

Use command_add_args to "hijack" Termdebug gdb launch commands. This
does not work with use_prompt: 1 though.

It also makes it difficult to add more arguments.

g:gdb_x_opt, g:gdb_x_opt_value can be used to load extra file
containing gdb command such as set up breakpoints etc.

This should be explained in the help.  When used in an example, there
should be a check if the variables exist.

It would be much nicer if all can be done with :Termdebug <host>:<port>, but that probbaly requries much more changes and it is
over my current capability.

It also conflicts with the current use of the argument, such as "core"
or "processID".  Adding a new command avoids confusion:
TermdebugRemote {host}:{port}

Would that work?
--
hundred-and-one symptoms of being an internet addict:
191. You rate eating establishments not by the quality of the food,
but by the availability of electrical outlets for your PowerBook.
/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///


Malformed URI "http://". ERROR In Request.php line 303

Laravel Version
11.0
PHP Version
8.2
Database Driver & Version
MySql
Description
I have updated the existing laravel 10 project to laravel 11.
and update all the dependency ( as per the document guide suggest here https://laravel.com/docs/11.x/upgrade ).
and i have run  composer update  command after removing the existing vendor folder and composer.lock file.
then this below issue arises.
Malformed URI "http://". ERROR In Request.php line 303
.env and config file also have the full url but i don't know why this error brake the project.

please let me know if any one know about this error or how to solve this.
Thanks in advance.
Steps To Reproduce

Change the existing dependency for laravel 10  to  laravel 11 as per the laravel 11 update guide  document suggest.
Run Composer update.

Hello, I doubt that it's a Laravel 11 issue.
This is what's happening during the Laravel bootstrapping, and more precisely during your composer update :
$uri = $app->make('config')->get('app.url', 'http://localhost');

And then your error happen later during the Request creation in symfony/http-foundation.
I suggest you to double-check the url property in config/app file.


Publish -sources artifacts

SnoopEE related artifacts under https://oss.sonatype.org/content/groups/public/eu/agilejava don't contain the regarding sources artfiacts. It'd be great to have them since IDEs now easily resolve and download the sources by clicking on the SnoopEE classes.

I'll look at it. The sources are published in the release to maven central, but it may have been left out in the snapshot builds.


Poor topbar design limits developers

I agree. Why it may work for most games, it being so limited in customizability is really making it hard to work with in more complex games. If a game has its own chat+health gui, and the scoreboard and backpack disabled, then the topbar becomes almost entirely useless space.
 I need something to do with it.

  What they need to add is
1 of course keep the zindexes of the menu and chat far above
But...  We need the ability to place user buttons such as shop menu guis etc on the topbar,  change the zindex to a lower calue so button guis are clickable on the topbar
If you need more details,  information on how to do such a thing,  etc,  message me.

And 2 a colorable topbar with the option to change opacity

Sorry for the spam but what this also ruins are some games with uset made topbats,  it pushes theor topbars down causing a partially obstructed view if we could click guis on the topbar (user created)  we could fix this (once again with shop guis,  spectating guis etc)

The developer forums is not necessarily open for the developers who make the most money. However being constructive and quantitative is definitely a criteria. Needless to say these posts don't help that case. This is the wrong place for design/product comments please visit http://www.roblox.com/Forum/ShowForum.aspx?ForumID=21 to post your suggestions and ideas!

@kyle-emmerich No longer a problem due to, http://devforum.roblox.com/t/hidable-topbar/22349

This issue was moved to GitBlox/Core-Scripts#5

tHAT me cUM in u LOL.

tHAT me cUM in u LOL.


EncryptionParameters::save gives nothing

filebuf fbParms;
fbParms.open("../SEALexample/serialization/encryptionParameters.txt", std::ios::out);
ostream osParms(&fbParms);
parms.EncryptionParameters::save(osParms);
fbParms.close();

I didn't get any errors, but encryptionParameters.txt kept being empty.
I use exactly the same format to serialize public key and ciphertext and they worked.

I was unable to replicate this issue. Can you please double-check your set-up. Additionally, please make sure to use std::ios::binary in addition to std::ios::out as the openmode, otherwise the data may become corrupted by unintended line-end conversions.

I was unable to replicate this issue. Can you please double-check your set-up. Additionally, please make sure to use std::ios::binary in addition to std::ios::out as the openmode, otherwise the data may become corrupted by unintended line-end conversions.

Thank you for your reply!
My code is like this:
class encryptionResult {
public:
    vector<array<Ciphertext, 3>> cipher;
    PublicKey public_key;
    EncryptionParameters parms;

    // Serialize all parameters to file
    void to_file() {
        // Serialize Ciphertext...

        // Serialize Public Key...

        // Serialize EncryptionParameters
        // out: serialization/encryptionParameters.txt
        **filebuf fbParms;
        fbParms.open("../SEALexample/serialization/encryptionParameters.txt", std::ios::out);
        ostream osParms(&fbParms);
        parms.EncryptionParameters::save(osParms);
        fbParms.close();**
    }
};

/** Encrypt image
 * @img image to be encrypted
 * @return encryption result
 */
encryptionResult encrypt(vector<array<int, 3>> img) {
    //Encryption...

    // Set parameters and create Context
    **EncryptionParameters parms(scheme_type::BFV);
    size_t poly_modulus_degree = 4096;
    parms.set_poly_modulus_degree(poly_modulus_degree);
    parms.set_coeff_modulus(CoeffModulus::BFVDefault(poly_modulus_degree));
    parms.set_plain_modulus(1024);**
    auto context = SEALContext::Create(parms);

    // Key generation
    // Create encryptor, evaluator and decryptor
    // Encryption

    encryptionResult output;
    output.cipher = encryptedMessage;
    output.public_key = public_key;
    output.parms = parms;

    return output;
}

int main() {
    char* filename = "../SEALexample/debug.bmp";
    vector<array<int, 3>> img = readBMP(filename);

    encryptionResult encrypted = encrypt(img);
    encrypted.to_file();

    return 0;
}

fbParms.close();

I don't have any relevant knowledge in C++, but shouldn't generally the uppermost stream be closed? In your case, that would be osParms.close(). Otherwise, the uppermost stream can still hold some no-yet-forwarded data in its cache. Pleaes ignore if that's not applicable here.

fbParms.close();

I don't have any relevant knowledge in C++, but shouldn't generally the uppermost stream be closed? In your case, that would be osParms.close(). Otherwise, the uppermost stream can still hold some no-yet-forwarded data in its cache. Pleaes ignore if that's not applicable here.

I am not a C++ expert myself as well :(, but the other two serialization (Ciphertext & Public Key) are in exactly the same structure as this one. If the problem indicated applies, the other two should output nothing as well.

If the problem indicated applies, the other two should output nothing as well.

Well, one would certainly hope that, but there's a reason why caching & timing issues are hard to track: inconsistent results. I'd really try osParms.close() instead of fbParms.close() or alternatively osParms.flush(). I guess that at least the public key is significantly bigger in size than encryption parameters and thus already triggering some flushing.

If the problem indicated applies, the other two should output nothing as well.

Well, one would certainly hope that, but there's a reason why caching & timing issues are hard to track: inconsistent results. I'd really try osParms.close() instead of fbParms.close() or alternatively osParms.flush(). I guess that at least the public key is significantly bigger in size than encryption parameters and thus already triggering some flushing.

I tried osParms.close() and osParms.flush().
osParms.close() doesn't exist, and osParms.flush() doesn't work. :-(

If you give EncryptionParameters some initial values (other than default values 0), is the file still empty? Were you able to load the same EncryptionParameters?

If you give EncryptionParameters some initial values (other than default values 0), is the file still empty? Were you able to load the same EncryptionParameters?

I think EncryptionParameters in my code shouldn't be empty.
Its value comes from here:
    // Generate encryptionResult obj for output
    encryptionResult output;
    output.cipher = encryptedMessage;
    output.public_key = public_key;
    output.parms = parms;

The function EncryptionParameters::save returns a number that is the total number of bytes written to stream or buffer. Can you check what this return value is? Is it zero?

The function EncryptionParameters::save returns a number that is the total number of bytes written to stream or buffer. Can you check what this return value is? Is it zero?

It may sound a little bit magical, but the problem disappeared just now for no reason.
I simply replace cout << "parms.EncryptionParameters::save(osParms) by cout << "TEST_NumBytesWritten: " << parms.EncryptionParameters::save(osParms) << endl;.

That's really strange! What compiler are you using? And what version exactly?

That's really strange! What compiler are you using? And what version exactly?

gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 (WSL)
cmake version 3.16.1
I guess it is resulted from some strange OS problem... probably not related to the code.


Have assertString.js provide the invalid type passed to validator.

As discussed in issue #894 I thought it would be useful for assertString.js to provide which invalid type was used as an input.
It checks explicitly for null, otherwise it will use typeof input. If typeof input === 'object' it will use the object's constructor's name (since 'object' alone is rarely helpful), though if that seems like overkill I can take that out.

Thanks for your contribution @DCtheTall 🎉
Please check my comments above.

@ProfNandaa ready for another look when you can.

Sure, will be on it in the morning. Thx
On Tue, Sep 25, 2018 at 1:47 AM Dylan Cutler<EMAIL_ADDRESS>wrote:

@ProfNandaa https://github.com/ProfNandaa ready for another look when
you can.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/chriso/validator.js/pull/895#issuecomment-424150524,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAP8kWWMXAsRr1GfBn1Vl03F3oukVrajks5ueWEIgaJpZM4Ws8QD
.

--
Sent from a tiny device while on the move.

@DCtheTall -- thanks for getting your first contribution in! 🎉


modify version up-lpg

Codecov Report
Patch and project coverage have no change.

Comparison is base (60c5386) 85.91% compared to head (f9d7984) 85.91%.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #425   +/-   ##
=======================================
  Coverage   85.91%   85.91%
=======================================
  Files         189      189
  Lines       24511    24511
=======================================
  Hits        21059    21059
  Misses       3452     3452

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


New Testimonial for Eddie Jaoude

Name
eddiejaoude
Title
Most helping and mind blowing person I saw in my life!!
Description
Eddie is super helpful to everyone, he had done a lot for the community, kindest and most amazing people I saw in my life❤
Because of Eddie, I stepped into Open Source, joined beautiful EddieHub Community💙

@eddiejaoude https://github.com/EddieHubCommunity/LinkFree/actions/runs/3845930154/jobs/6550649453

Oh thanks @schmelto - we need to improve that 🤔  ideas welcome 😂

Thank you @MadhuSaini22 for the amazing testimonial ❤