[S3] Select is broken on large-ish files

Reproducible example; https://github.com/aws/aws-sdk-net/issues/1296#issuecomment-493769504
Expected Behavior
The count should be correct:-
Expected...
Key: [BCD, 20933]
Key: [DEF, 13528]
Key: [FGH, 12188]
Key: [HIJ, 16105]
Key: [JKL, 21117]
Key: [LMN, 13560]
Key: [NOP, 18547]

Current Behavior
The count is wrong for result set:-
Actual...
Key: [BCD, 20933]
Key: [DEF, 13527]
Key: [FGH, 12187]
Key: [HIJ, 16105]
Key: [JKL, 21117]
Key: [LMN, 13560]
Key: [NOP, 18545]

I also get malformed data returned, every line should be in the s._1, s._2 format according to the query:-
bad line: DE
bad key: F
bad line: 3
bad line: NO
bad key: P
bad line: 3
bad line: NO
bad key: P
bad line: 3
bad line: FG
bad key: H

Possible Solution
Haven't investigated further, just can consistently reproduce broken behaviour
Steps to Reproduce (for bugs)
See above
Context

Your Environment

AWSSDK.Core version used: <IP_ADDRESS>
Service assembly and version used: <IP_ADDRESS>
Operating System and version: Windows 10 Build 1809
Visual Studio version: 15.9.11
Targeted .NET platform: 4.5.2

I've only been able to reproduce this on "large" files. I don't know where the threshold is, but on smaller files (< 10 MB). My results are consistent correct.

I also haven't been able to ascertain whether this is an SDK issue or a service issue.

Slightly more concise reproduction:-
public static void Main(string[] args)
{
	var client = new AmazonS3Client(new BasicAWSCredentials(_accessKey, _secretKey), RegionEndpoint.EUWest1);

	Console.WriteLine("Generating file...");
	var expectedCount = GenerateFile("BCD", "DEF", "FGH", "HIJ", "JKL", "LMN", "NOP");

	Console.WriteLine("Uploading file...");
	UploadFileToS3(client);

	Console.WriteLine("Selecting file from S3 to get count..");
	SelectFile(client, expectedCount, "BCD", "DEF", "FGH", "HIJ", "JKL", "LMN", "NOP");
}

private static void UploadFileToS3(AmazonS3Client client)
{
	client.PutObject(new PutObjectRequest
	{
		Key = _fileKey,
		BucketName = _bucketName,
		InputStream = File.OpenRead(_localPath)
	});
}

private static void SelectFile(AmazonS3Client client, IDictionary<string, int> expected, params string[] keys)
{
	var keys_in = string.Concat("'", string.Join("', '", keys), "'");

	var response = client.SelectObjectContent(new SelectObjectContentRequest
	{
		Key = _fileKey,
		Bucket = _bucketName,
		ExpressionType = ExpressionType.SQL,
		Expression = $"select s._1, s._2 from S3Object s WHERE s._1 IN ({keys_in}) AND s._2 = '3'; ",
		InputSerialization = new InputSerialization
		{
			CSV = new CSVInput()
		},
		OutputSerialization = new OutputSerialization
		{
			CSV = new CSVOutput()
		}
	});

	var actual = new Dictionary<string, int>();

	foreach (var key in keys)
	{
		actual[key] = 0;
	}

	using (var stream = response.Payload)
	{
		foreach (var ev in stream)
		{
			if (ev is RecordsEvent records)
			{
				using (var reader = new StreamReader(records.Payload, Encoding.UTF8))
				{
					while (reader.EndOfStream == false)
					{
						var line = reader.ReadLine();

						if (line.Contains(","))
						{
							var tokens = line.Split(',');
							var key = tokens[0];

							if (actual.ContainsKey(key))
							{
								actual[key]++;
							}
							else
							{
								Console.WriteLine("bad key: " + key);
							}
						}
						else
						{
							Console.WriteLine("bad line: " + line);
						}
					}
				}
			}
		}
	}

	Console.WriteLine("Actual...");
	foreach (var key in keys)
	{
		var expectedValue = expected[key];
		var actualValue = actual[key];
		Console.WriteLine($"Key: {key} Actual: {actualValue} Expected: {expectedValue} Different? {actualValue != expectedValue}");
	}
}

private static IDictionary<string, int> GenerateFile(params string[] keys)
{
	var random = new Random(252028800);

	var dict = new Dictionary<string, int>
	{
		["ABC"] = random.Next(100000, 200000),
		["BCD"] = random.Next(100000, 200000),
		["CDE"] = random.Next(100000, 200000),
		["DEF"] = random.Next(100000, 200000),
		["EFG"] = random.Next(100000, 200000),
		["FGH"] = random.Next(100000, 200000),
		["GHI"] = random.Next(100000, 200000),
		["HIJ"] = random.Next(100000, 200000),
		["IJK"] = random.Next(100000, 200000),
		["JKL"] = random.Next(100000, 200000),
		["KLM"] = random.Next(100000, 200000),
		["LMN"] = random.Next(100000, 200000),
		["MNO"] = random.Next(100000, 200000),
		["NOP"] = random.Next(100000, 200000),
		["OPQ"] = random.Next(100000, 200000)
	};

	var terms = new[] {12, 24, 36, 48, 60};
	var mileages = new[] {6000, 10000, 15000, 20000, 25000, 30000, 40000, 50000};
	var list = new List<string>();

	var keysWeCareAbout = new Dictionary<string, int>();

	foreach (var key in keys)
	{
		keysWeCareAbout[key] = 0;
	}

	foreach (var pair in dict)
	{
		for (int i = 0; i < pair.Value; i++)
		{
			var elementId = random.Next(1, 10);
			var next = random.Next(100000, 999999);
			var term = terms[random.Next(0, terms.Length)];
			var mileage = mileages[random.Next(0, mileages.Length)];

			if (keysWeCareAbout.ContainsKey(pair.Key) && elementId == 3)
			{
				keysWeCareAbout[pair.Key]++;
			}
			var line = $"{pair.Key},{elementId},{next},{term},{mileage},{random.Next(1, 100)}.{random.Next(1, 99)},,";
			list.Add(line);
		}
	}

	File.WriteAllLines(_localPath, list.OrderBy(x => random.Next()), Encoding.UTF8);

	return keysWeCareAbout;
}

It looks like something is malforming the lines:-

It looks like something is malforming the lines:-
bad line: DE
bad key: F
bad line: 3
bad line: NO
bad key: P
bad line: 3
bad line: NO
bad key: P
bad line: 3
bad line: FG
bad key: H

If we join the "bad" output we get this:-
DEF3NOP3NOP3FGH

And if you look at what I'm missing from the counts:-
Key: BCD Actual: 20933 Expected: 20933 Different? False
Key: DEF Actual: 13527 Expected: 13528 Different? True
Key: FGH Actual: 12187 Expected: 12188 Different? True
Key: HIJ Actual: 16105 Expected: 16105 Different? False
Key: JKL Actual: 21117 Expected: 21117 Different? False
Key: LMN Actual: 13560 Expected: 13560 Different? False
Key: NOP Actual: 18545 Expected: 18547 Different? True

I'm missing one DEF, one FGH, and two NOP.
Still don't know if this is a service issue or SDK issue.

If you hit S3 Select directly in Postman (bucketName.s3.amazonaws.com/fileKey?select&select-type=2), using this request:-
<SelectRequest>
    <Expression>select s._1, s._2 from S3Object s WHERE s._1 IN ('BCD', 'DEF', 'FGH', 'HIJ', 'JKL', 'LMN', 'NOP') AND s._2 = '3'</Expression>
    <ExpressionType>SQL</ExpressionType>
    <InputSerialization>
        <CSV>
            <FieldDelimiter>,</FieldDelimiter>
        </CSV>
    </InputSerialization>
    <OutputSerialization>
        <CSV>
            <FieldDelimiter>,</FieldDelimiter>
        </CSV>
    </OutputSerialization>
</SelectRequest>

You'll see the records event is chopping up the stream:-
JKL,3
NO– OÅ  þM   UñØžƒ
:message-type event:event-type Records
:content-type application/octet-streamP,3
LMN,3
JKL,3

If you look carefully it has chopped up NOP,3 by inserting the records events in between O and P. This would explain the bad line / bad key output in my earlier comment.

If I change the SQL statement to be:-
Expression = $"SELECT s._1, s._2 from S3Object s WHERE s._1 IN ({keys_in}) AND s._2 = '3' AND s._4 = '48' AND s._5 = '30000' ; ",

And the if statement to be:-
if (keysWeCareAbout.ContainsKey(pair.Key) && elementId == 3 && term == 48 && mileage == 30000)

Then everything is perfect:-
Key: BCD Actual: 481 Expected: 481 Different? False
Key: DEF Actual: 350 Expected: 350 Different? False
Key: FGH Actual: 305 Expected: 305 Different? False
Key: HIJ Actual: 382 Expected: 382 Different? False
Key: JKL Actual: 515 Expected: 515 Different? False
Key: LMN Actual: 350 Expected: 350 Different? False
Key: NOP Actual: 430 Expected: 430 Different? False

I don't get any bad output.
And if I change the XML request to be:-
<Expression>select s._1, s._2 from S3Object s WHERE s._1 IN ('BCD', 'DEF', 'FGH', 'HIJ', 'JKL', 'LMN', 'NOP') AND s._2 = '3' and s._4 = '48' and s._5 = '30000'</Expression>

Then the records event is still there but happens to slice up the stream correctly:-
JKL,3
FGH,3
v?
weUX3BT
:message-typeevent:event-typeRecords
:content-typeapplication/octet-streamJKL,3
DEF,3
JKL,3

I still can not confirm if the S3 Select Service is at fault here or if the SDK is reading the incoming stream incorrectly in some scenarios.

I've managed to narrow this down to a test case that returns bad results 48% of the time. I've updated my https://github.com/aws/aws-sdk-net/issues/1296#issuecomment-493769504 containing the relevant code.
Snippet of output:-
Index: 4
Selecting file from S3 to get count..
Got record event
bad line: DE
Got record event
bad key: F
Key: DEF Actual: 1276 Expected: 1277 Match? False
Not good

Index: 5
Selecting file from S3 to get count..
Got record event
bad line: DE
Got record event
bad key: F
Key: DEF Actual: 1276 Expected: 1277 Match? False
Not good

Index: 6
Selecting file from S3 to get count..
Got record event
Got record event
All good

Index: 7
Selecting file from S3 to get count..
Got record event
bad line: DE
Got record event
bad key: F
Key: DEF Actual: 1276 Expected: 1277 Match? False
Not good

Index: 8
Selecting file from S3 to get count..
Got record event
Got record event
All good

I still haven't managed to confirm if it is a service or SDK problem.

@sstevenkang is there any other information that would be helpful in getting this resolved?

I'm taking a look

The problem is we have a usability issue with the S3 select API in the SDK that we need to address.
When you are reading the RecordsEvent objects from the Payload, S3 makes no guarantees that it contains only full csv lines. Sometimes a csv line will be split over multiple RecordEvents. So you need to make sure you are reading from the RecordsEvent stream till you have a full csv line before doing any processing on it. I updated your reader code below to keep reading through multiple RecordsEvent till a newline was read and then do your processing. Once I did that I was able to successfully do the 100 iterations in your test code.
The SDK should have done a better job of abstracting this but I think when we originally designed the API we didn't take into account getting partial lines in a RecordsEvent. I'll talk with the rest of the team to see how we can improve this feature in the SDK.
using (var stream = response.Payload)
{
    StringBuilder fullCsvRecord = new StringBuilder();
    foreach (var ev in stream)
    {
        if (ev is RecordsEvent records)
        {
            Console.WriteLine("Got record event");
            using (var reader = new StreamReader(records.Payload, Encoding.UTF8))
            {
                while (reader.EndOfStream == false)
                {
                    var c = (char)reader.Read();
                    if(c == '\n')
                    {
                        var line = fullCsvRecord.ToString();
                        fullCsvRecord.Clear();

                        if (line.Contains(","))
                        {
                            var tokens = line.Split(',');
                            var key = tokens[0];

                            if (actual.ContainsKey(key))
                            {
                                actual[key]++;
                            }
                            else
                            {
                                Console.WriteLine("bad key: " + key);
                            }
                        }
                        else
                        {
                            Console.WriteLine("bad line: " + line);
                        }
                    }
                    else
                    {
                        fullCsvRecord.Append(c);
                    }
                }
            }
        }
    }
}

Ahhh I see. Is it possible to update the documentation to reflect that? Or did I miss that entirely?
I think it would be awesome if the .NET API could do the heavy lifting for me. In my head if a declare s._1, s._2 then I would expect the data to be returned in that fashion line by line. It would also be nice if it could handle different types of line endings for me (e.g \n, \r\n).
Does the API guarantee order? Cause the other option to is ReadToEnd all the RecordsEvent into one StringBuilder and then do the processing.
Would the event driven API suffer from the same issue? I used this blog post as a base line example.
I'm happy for this issue to be closed, if need be.
Thanks for helping me :) :+1:

I'll talk with the rest of the team to see how we can improve this feature in the SDK.

Any update on what, if anything was decided @normj ?

Just going through and sorting out my open issues; any update on this @normj ?
Thanks,
Indy

Hi @indy-singh,
Good afternoon.
I was going through the list of open issues and came across this one. From @normj comments, looks like there is a workaround available. Please advise if you would like to convert this guidance question into a feature-request (I'm not sure on how sooner this could b prioritized) or if we could close this issue.
Thanks,
Ashish

Hi @ashishdhingra ,
I would absolutely like the SDK to support this as a feature, so yes please turn this guidance question into a feature-request.
Thanks,
Indy


when i run bash run.sh --stage 4 --stop-stage 4 ,got error:

Traceback (most recent call last):
File "wenet/bin/train.py", line 209, in
executor.train(model, optimizer, scheduler, train_data_loader, device,
File "/media/dayu/D/nlp/wenet/examples/aishell/s0/wenet/utils/executor.py", line 35, in train
loss, loss_att, loss_ctc = model(feats, feats_lengths, target,
File "/home/dayu/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/media/dayu/D/nlp/wenet/examples/aishell/s0/wenet/transformer/asr_model.py", line 89, in forward
encoder_out, encoder_mask = self.encoder(speech, speech_lengths)
File "/home/dayu/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/media/dayu/D/nlp/wenet/examples/aishell/s0/wenet/transformer/encoder.py", line 133, in forward
masks = ~make_pad_mask(xs_lens).unsqueeze(1)  # (B, 1, L)
File "/media/dayu/D/nlp/wenet/examples/aishell/s0/wenet/utils/mask.py", line 140, in make_pad_mask
max_len = int(lengths.max().item())
RuntimeError: CUDA error: no kernel image is available for execution on the device
my evn is ubuntu20.04,cuda11.1 torch17.1

please follow the readme to setup your env.
install miniconda
conda create -n wenet python=3.8
conda activate wenet
pip install -r requirements.txt
conda install pytorch==1.6.0 cudatoolkit=10.1 torchaudio -c pytorch


Refactoring CSS Code for Improved Readability and Organization ✨

Overview
During this refactoring process, I made several changes to the CSS code in order to improve its readability, organization, and maintainability. The primary goal was to make the code more structured and easier to understand, while also ensuring consistency in naming conventions and indentation.
Changes Made

Grouping Related CSS Rules: I grouped related CSS rules together to improve organization and make it easier to locate styles for specific sections.

Adding Comments: I added comments to describe each section of the code, making it clearer what each part of the CSS is responsible for.

Consistent Indentation: I ensured consistent indentation throughout the code, which helps in visually understanding the CSS hierarchy.

Organizing Media Queries: The media queries were properly organized based on screen size, making it easier to manage different device-specific styles.

Consolidating Repeated Styles: I identified and consolidated repeated styles to avoid redundancy and make the code more concise.

Improved Naming Conventions: I reviewed class and ID names, using descriptive names where possible to enhance code clarity.

Before Refactoring:
[Original CSS code ]

#contact {
  background-image: url("/assets/img/background/contact.webp");
  width: 100%;
  height: 100%;
  color: white;
  padding-top: 100px;
  background-repeat: no-repeat;
  background-size: cover;
}

/* ... other styles ... */

.contactForm {
  padding: 30px 10px;
  background-color: rgb(228, 227, 227);
  border-radius: 10px;
  box-shadow: rgb(1, 2, 34) 0px 10px 15px;
}

/* ... other styles ... */

.mobile-nav-toggle {
  color: var(--background-white);
  font-size: 28px;
  cursor: pointer;
  display: none;
  line-height: 0;
  transition: 0.5s;
  padding-right: 25px;
}

/* ... other styles ... */

After Refactoring:
[Refactored CSS code }

/* Contact section */
#contact {
  background-image: url("/assets/img/background/contact.webp");
  width: 100%;
  height: 100%;
  color: white;
  padding-top: 100px;
  background-repeat: no-repeat;
  background-size: cover;
}

#contact .contBox {
  background-color: #000e47b5;
  padding: 50px;
  width: 95%;
}

/* ... other styles ... */

.contactForm {
  padding: 30px 10px;
  background-color: rgb(228, 227, 227);
  border-radius: 10px;
  box-shadow: rgb(1, 2, 34) 0px 10px 15px;
}

/* ... other styles ... */

/* Mobile Navigation */
.mobile-nav-toggle {
  color: var(--background-white);
  font-size: 28px;
  cursor: pointer;
  display: none;
  line-height: 0;
  transition: 0.5s;
  padding-right: 25px;
}

.mobile-nav-toggle.bi-x {
  color: var(--background-white);
}

/* ... other styles ... */

Conclusion
The refactored CSS code should be more organized, readable, and easier to maintain compared to the original version. The changes made aim to improve the development experience and ensure that future updates and modifications can be done with greater ease. If you have any questions or concerns about the changes, feel free to reach out!

Hey @Vaishnavi-Patil2211  I am done with the code please assign me the issue i will make pr ✨


Update mod_auth_openidc.c

fixes #447

can you add something to the ChangeLog and update the AUTHORS file?

can you add something to the ChangeLog and update the AUTHORS file?

done


Move NetworkPolicy handlers to their own package

@ibuildthecloud requested this change.
Checklist

[x] The title of this PR would make a good line in Acorn's Release Note's Changelog
[ ] The title of this PR ends with a link to the main issue being address in parentheses, like: This is a title (#1216). Here's an example
[x] All relevant issues are referenced in the PR description. NOTE: don't use GitHub keywords that auto-close issues
[x] Commits follow contributing guidance
[x] Automated tests added to cover the changes. If tests couldn't be added, an explanation is provided in the Verification and Testing section
[x] Changes to user-facing functionality, API, CLI, and upgrade impacts are clearly called out in PR description
[ ] PR has at least two approvals before merging (or a reasonable exception, like it's just a docs change)

Looks like the testdata didn't get moved over the process


Dataflow pipeline with added steps

Added these steps to the pipeline:

Parsing the csv. Requires to use a folder constants.py (which will be shared with TF).
Splitting the data into train/eval: f I am correct, the current TF code uses the test set as validation. even if it is not an important change, it is just to illustrate CMLE functionalities.
Outputs back to csv.

Next steps will be:

Updating README
Integrating with TF.

@cfezequiel @dan-anghel @gogasca @dizcology @thedriftofwords @happyhuman


Firestore - Project Id

This is just a general question about the Firestore Library (Google.Cloud.Firestore 2.3.1).  When creating a new FirestoreDb object, I keep receiving an error stating that "ProjectId cannot be null".  However, I'm wondering why a Project Id is required when the project Id is also included in the service account key file.  Is there a way to tell the library to use the value in the key file, instead of passing it in to the Create method?
FirestoreDb.Create()

The credential used to connect isn't always a service account.
When running on GCP, the project ID is auto-detected from the metadata server, but there's no equivalent when the deployment isn't on GCP.
If you know that your code will always have a service account key, you could get that credential yourself using GoogleCredential.GetApplicationDefault(), use the UnderlyingCredential property and cast it to ServiceAccountCredential, then extract the project ID from that, but we don't do that in the library at the moment.

Thank you.


(1972) Send email to new users

This adds Bull to allow us to queue and process jobs, as well as using Bull to send emails via GOV.UK Notify.
I've taken the approach we have had in other projects and added a very simple template to GOV.UK Notify, with (( subject )) and (( body )) variables in like so:

This allows us to use the same template across the project, as well as keeping our email body content tightly coupled to the code and version controlled.

Able to add dump.rdb to .gitignore as part of this?


Define ssh key distribution process

This is part of SSDF PO 5.1  and has been flagged elsewhere. We should have a process for distributing ssh keys wherever they are used to avoid having to bypass the checks on first connect. For example:

Nagios
Jenkins
Infrastructure team
This could be via explicitly adding the host key into the inventory file.

Note: Jenkins won't accept an ssh-rsa or ssh-dss key and will therefore requre the contents of ssh_host_ecdsa_key.pub or ssh_host_ed25519_key.pub to be used in the agent configuration

@Haroon-Khel is going to look at implementing the changes for the other jenkins nodes which we connect to over ssh (The test- ones) and look at ensuring that any nodes created as part of the dockerstatic node automation will support this too.

@sxa do we have an existing document for this?, otherwise I'll write a document with our current policy and processes for handling these things.

Sounds good thanks - we have no such document currently. This is very much a "let's define and document the process" issue :-)

All of the static docker containers have been restarted and are connected to jenkins with host key verification, except for the nodes on https://ci.adoptium.net/label/hw.dockerhost.dockerhost-skytap-ubuntu2204-x64-1/. I think that dockerhost is down as are its nodes

New document for SSH key management is here.. https://docs.google.com/document/d/1ltGOhmRKkcN-CvafiNRDZwGKCyUC8xb8lCLWBh1VJEI/edit?usp=sharing
@sxa would you mind reviewing, I think that means this issue might be complete.

I've added a few comments. On making them, I'm thinking that it might make sense to have separate recommendations for infrastructure team access (where root access is typically granted to the whole team) and for people who have "temporary" access to a user account on a machine, which I wouldn't necessarily hold to such high standards (e.g. I wouldn't mandate a particular cipher or passphrase for such an account, which would raise the barrier for access)

Updates have been made, following review comments, 2 new sections added , regarding temporary/additional access and jenkins host key management.

Closing, as I believe this is completed.

@steelhead31 Has that document been put anywhere or is it still just in google doc? It looks like it's restricted access at the moment.

@sxa, I think the google doc is it at the moment, Im sure I asked if there was a suitable public space... let me know where if such a place exists! :)


Intel NUC 7thGen Video Acceleration issues

been having a bit of an issue, and i'm not sure if this issue belong here in the OS or if its a Docker problem.
Steps to reproduce issue

Flash the resinOS provided by the resin dashboard (2.0.9+rev1 as the time of this writing) .
Using Resin/Electron as a base i make the necessary adjustments in the Dockerfile.template to make video output work :

use docker image base resin/intel-nuc-node:8
remove the line: xserver-xorg-video-fbdev \
add RUN echo deb http://http.debian.net/debian jessie-backports main contrib non-free > /etc/apt/sources.list.d/jessie-backports.list
add RUN apt-get update
add RUN sudo apt-get install --target-release jessie-backports linux-image-amd64 firmware-misc-nonfree xserver-xorg-video-intel libegl1-mesa libgl1-mesa-glx libgl1-mesa-dri
change npm cache clean to npm cache verify

Put some CSS 3d animation or video on the electron app for video acceleration testing
Push the app to the repo and let the NUC provision itself.

Expected result

Webview to start with video acceleration.

Actual result

App crashes with different errors, among them some of the like:

[intel_init_bufmgr] Error initializing buffer manager.
xinit: connection to X server lost
[WARNING:audio_manager.cc] Multiple instances ofAudioManager detected

Other information

This was just for testing but i also added in one instance intel_iommu=igfx_off in the grub.cfg file. (needless to say it didn't help).
Disabling hardware acceleration & webgl in electron config just makes it so the app is visible until you interact with it and some animation happens then the app crashes.

@curcuz do you have any input here about electronjs?


Blueprint of a test module for phospho

Add a basic implementation to do backtesting with an agent with the following syntax
# phospho_test.py
phospho_test = phospho.PhosphoTest()

 @phospho_test.test
 def test_santa(**inputs):
     agent = Agent()
     return agent.answer(**inputs)

 phospho_test.run()

python phospho_test.py

To debug : pandas mode


update documenation for version 0.3.0

Description
Before releasing version 0.3.0 there MUST be a complete update of the documentation. Because of my limited available time I can not update the documentation of each of the current changes, so I decied to make a complete update at the end before releasing 0.3.0, to save some time.

futher updates will be done in #126


dogg3rz insert node doesn't update mtimes when new node is inserted

The scenario is as follows:

create a new dataset
insert a node (into default graph)

The state of the json-ld doc are as follows for each step. Notice that the number of mtimes entries stay the same after the insert node operation.
after create dataset:
{
   "__dgrz_mtimes__" : {
      "9dcf97a184f32623d11a73124ceb99a5709b083721e878a16d78f596718ba7b2" :<PHONE_NUMBER>707369000,
      "25dfd29c09617dcc9852281c030e5b3037a338a4712a42a21c907f259c6412a0" :<PHONE_NUMBER>707369000
   },
   "@graph" : [],
   "@context" : {}
}

after insert node(default-graph):
{
   "@context" : {},
   "@graph" : [
      {
         "@id" : "_:b4879502-17b5-4418-80c0-3f7df44dbe52",
         "attr1" : "value1"
      }
   ],
   "__dgrz_mtimes__" : {
      "25dfd29c09617dcc9852281c030e5b3037a338a4712a42a21c907f259c6412a0" :<PHONE_NUMBER>707369000,
      "9dcf97a184f32623d11a73124ceb99a5709b083721e878a16d78f596718ba7b2" :<PHONE_NUMBER>707369000
   }
}

@bhallamp I see see the same issue where only 2 entries are in dgrz_mtimes  after inserting a node. Expect 3.

fixed:
pbhallam@xxx ds1 % cat .document.jsonld | jq
{
"@context": {},
"@graph": [],
"dgrz_mtimes": {
"25dfd29c09617dcc9852281c030e5b3037a338a4712a42a21c907f259c6412a0":<PHONE_NUMBER>624102000,
"9dcf97a184f32623d11a73124ceb99a5709b083721e878a16d78f596718ba7b2":<PHONE_NUMBER>624102000
}
}
pbhallam@xxx ds1 % cat .document.jsonld | jq
{
"@context": {},
"@graph": [
{
"@id": "_:9aa7a260-a542-4390-b357-35babad46d79",
"attr1": "val1"
}
],
"dgrz_mtimes": {
"25dfd29c09617dcc9852281c030e5b3037a338a4712a42a21c907f259c6412a0":<PHONE_NUMBER>624102000,
"50fc7b7a344d63b245bb09ee843a2b88728ff178731e449105d3b3c42f092ee3":<PHONE_NUMBER>049942000,
"9dcf97a184f32623d11a73124ceb99a5709b083721e878a16d78f596718ba7b2":<PHONE_NUMBER>624102000
}
}

verified as fixed


#39 #44 Add brief kotlin style guide description with reference to of official Kotlin style guide

This pull adds short Kotlin convention and points to official one which is quite good. Official convention also describes brackets around loops and code-blocks (#44)

Please add paragraph for brackets around loops and code-blocks and link there to official guide.


import: add a hash check for http(s) imports

This builds on top of 60cb730
Before this commit it only checked  HTTP header "X-Checksum-Sha256" with the hash provided in stacker.yaml
Now it also verifies the sha256 of the newly downloaded file on top of existing checks
Also resolves ~part1 of #206

One small nit, but other than that this looks great, thank you!


Add a global redjubjub verification context with fallback.

This moves the redjubjub module into a new primitives module to prepare for adding a groth16 verifier, and provides a global verification service wrapped in a once_cell::Lazy.

Clippy warns on unused statics because we don't ever do any verification of redjubjub signatures.

Clippy warns on unused statics because we don't ever do any verification of redjubjub signatures.

Can we add allow(dead_code) in the appropriate place, until these verifiers are used?
Otherwise, we'll get a bunch of spurious clippy warnings on every build and PR after merge.

Yes, I did this as a small piece of preparatory work for adding a primitives::groth16 module.  The constants are in the parent module so they can be reused there.
To handle the dead code warnings I would suggest adding an #[allow(dead_code)] to the entire primitives module for now.


[controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node rancher-controller1: "rancher-controller1" not found]]

Simplest deployment - single node in VMware.
Define cloud vSphere, node template, using boot2docker ISO legacy, with latest image (left black)
Getting error:
[controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node rancher-controller1: "rancher-controller1" not found]]
Who is producing this error? Is it coming from node itself? from rancher? the IP address is known to rancher... because I see it on the UI.
Environment information
Rancher | v2.4.5
User Interface | v2.4.28
Helm | v2.16.8-rancher1
Machine | v0.15.0-rancher43

Anyone? I am desperate  ...

Please supply full logging output from the rancher/rancher container.

I figured it out... it seems that if nodes cross network broadcast domain (VLANs in our case), even if routable, rancher fails. Wonder why that is... probably some broadcast related implementation.

I have the same problem, how did you solved it?

All deployments need to be in the same broadcast domain... In another word,
your rancher controller and nodes is on the same subnet.
On Mon, Nov 9, 2020 at 9:06 AM zopar<EMAIL_ADDRESS>wrote:

I have the same problem, how did you solved it?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/28235#issuecomment-724146426,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB2FAFMOI4YE5AIB47NH24DSPAOTFANCNFSM4PVAB2LA
.


In example 4, "arg2" undeclared in the neco_main

Hello, I'm trying each of the examples and found a small mistake in example 4.
    int *arg1 = malloc(sizeof(int));
    *arg1 = 1;

    neco_start(coroutine, 5, &arg0, arg1, &(int){2}, NULL, "hello world");
    free(arg2);

free(arg2) should be free(arg1).
Regards

Fixed. Thanks for reporting!


DILE_VT/Quirks: Implement QUIRK_DILE_VT_NO_FREEZE_CAPTURE

Do not wait for DILE_VT_SetVideoFrameOutputDeviceState: FREEZED in capture_acquire_frame

It is not compatible with every TV, hence implementing it as a quirk
Yields alot higher FPS for me (went from 8 fps -> 50 fps)

Would suggest to do the README update as a seperate commit?


[Bug] tensorflow error : tensorflow/stream_executor/cuda/cuda_driver.cc:404] Check failed

Description
python crashed when got below error message.

Step to reproduce

Installed using pip install spleeter
Run as spleeter separate -i spleeter/audio_example.mp3 -p spleeter:2stems -o output
Got tensorflow/stream_executor/cuda/cuda_driver.cc:404] Check failed: CUDA_SUCCESS == cuDevicePrimaryCtxGetState(device, &former_primary_context_flags, &former_primary_context_is_active) (0 vs. 303) error

Output
tensorflow/stream_executor/cuda/cuda_driver.cc:404] Check failed: CUDA_SUCCESS == cuDevicePrimaryCtxGetState(device, &former_primary_context_flags, &former_primary_context_is_active) (0 vs. 303)

Environment

OS
Windows

Installation type
pip

RAM available
3694MB Go

Hardware spec
CPU

Additional context

after readed spleeter source code, I finally soloved this problem by run:
spleeter separate -i spleeter/audio_example.mp3 -p spleeter:2stems -o output -B librosa


[feature request] Add noise cancel control

Thank you for creating this helpful tool. I wonder if it's possible to add noise control to this workflow, like typingapc adaptive to switch to the adaptive mode.

Got it, thanks for the reply
Sincerely,
Xiao Meng
On Sep 30, 2024 at 23:33:50, G. @.***> wrote:

this Workflow is using blueutil https://github.com/toy/blueutil behind
the scenes to connect/disconnect bluetooth devices. blueutil doesn't have
anything related to audio devices in particular, and really is about
connecting. alfred-airpodspro-battery
https://github.com/godbout/alfred-airpodspro-battery would be a better
target but then it'd be about controller your AirPods rather than reading
data, so it's kind of another Workflow/work altogether. not seeing myself
doing this.
—
Reply to this email directly, view it on GitHub
https://github.com/godbout/alfred-airpodspro-connector/issues/13#issuecomment-2383537728,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADDDVCF54NIADEULS4HBLTZZFVN5AVCNFSM6AAAAABPCKCHT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBTGUZTONZSHA
.
You are receiving this because you authored the thread.Message ID:
@.***>

不客气。maybe try to post on the battery Workflow and i'll think about it more 哈哈哈哈哈哈哈

That's a great idea, I have added the new issue.
Besides, I read your website and projects, they are really inspiring (the about page), and thoughtful (sleeplessmind.info), it's really nice to get to know you.

Besides, I read your website and projects, they are really inspiring (the about page), and thoughtful (sleeplessmind.info), it's really nice to get to know you.

🙏🙏🙏 sleeplessmind.info is super focused on writings. https://glmb.today is where you get most of the day to day (month to month...) updates tho 😅 谢谢你！也非常好认识你。你在中国哪儿？

seeing that sleeplessmind.info needs some update in the about part. nice. will do.


Support skipping output generation based on API specification checksum

Opt-in feature to allow checking for source API specification checksum. If output file exists and both source checksum and toolchain version matches in the file, generation is skipped altogether. Requires nswag.json to have "checksumCacheEnabled": true for TS/CS generation configuration for checks to kick in.
If user has created custom templates that can change on disk this feature isn't for them, but if only built-in ones are used the cache should be safe.

Thanks for the PR.


Add speech enhancement models

Reference Issue
Adding speech enhancement model
Any other comments?
We need to discuss how this can fit in the documentation and the readme. I added it to the pre-trained model section and introduced in the README. I'm open for suggestions here (e.g. should we incorporate details in training.md etc or just refer to https://sigsep.github.io/open-unmix/se ?

ping @TE-StefanUhlich

@aliutkus this is ready to merge, can you please review the doc changes?

ping @aliutkus

ping @aliutkus

will do soon, sorry


Need smalltalk logic to call thread safe library GciTsNbLogout

The Non-blocking logout for a thread safe gci library would be handy because it doesn't wait for a reply from the server. Currently, the GemStoneFFI smalltalk package only implements gciTsLogout:gsError: and not the non-blocking variant  as far as I can tell.
Currently, when using RSR through the gci library, if a gcli logout gets in trouble, the pharo image will hang.
See:
GciThreadSafeInterface>>#logout
GciThreadSafeInterface>>#logoutNoError
GciThreadSafeInterface>>#gciTsLogout:gsError:

Code added to v.1.1 . See GsSession>>logoutNbNoError


Flavor 'nativescript-socket.io' has unknown dimension 'nativescriptsocketio'

I'm getting the following error when attempting to build and/or run a bare-bones android nativescript app under Linux 5.0.4:
$ uname a
Linux bkuri-laptop 5.0.4-1-ck-skylake #1 SMP PREEMPT Sat Mar 23 19:56:08 EDT 2019 x86_64 GNU/Linux

$ tns build android --bundle
Skipping prepare.
Building project...
Gradle build...
         + setting applicationId
         + applying user-defined configuration from /home/bk/source/blix/app/App_Resources/Android/app.gradle
Using support library version 28.0.0
         + adding nativescript runtime package dependency: nativescript-optimized-with-inspector
         + adding aar plugin dependency: /home/bk/source/blix/node_modules/nativescript-ui-sidedrawer/platforms/android/TNSSideDrawer-release.aar
         + adding aar plugin dependency: /home/bk/source/blix/node_modules/nativescript-ui-core/platforms/android/TNSCore-release.aar
         + adding aar plugin dependency: /home/bk/source/blix/node_modules/tns-core-modules-widgets/platforms/android/widgets-release.aar

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring project ':app'.
> Flavor 'nativescript-socket.io' has unknown dimension 'nativescriptsocketio'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 0s
Command ./gradlew failed with exit code 1

Before that error appeared I had to fix another flavor-related error by adding the flavorDimensions line to app/App_Resources/Android/app.gradle (not sure if it's related):
// Add your native dependencies here:

android {
  flavorDimensions 'default'

  defaultConfig {
    generatedDensities = []
    applicationId = "com.blix.app"
  }

  aaptOptions {
    additionalParameters "--no-version-vectors"
  }
}

Vote up.

try unmounting the device and then remouting it. 'tns platform remove android' then tns platform add android
this fixed the issue for me


Implement WifiAccessPointSetPhyType

Type

[ ] Bug fix
[ ] Feature addition
[X] Feature update
[ ] Documentation
[ ] Build Infrastructure

Side Effects

[ ] Breaking change
[ ] Non-functional change

Goals
Describe what the PR intends to achieve. If the change is a new feature, describe what it is. If the change is a bug fix, reference the issue being fixed. Provide any additional context and motivation for making the change. Eg.

Simplify maintenance of the event loop. A well-known open source library can be used in place of the custom written code, making it easier to use, understand, and debug.

Technical Details
Describe how the goals of the PR were achieved. Eg.

Replace custom event loop code with libevent.

Test Results
Describe which tests were carried out and a summary of their results. Eg.

Verified Near Object session completed successfully with the following configurations:

[X] host as controlee
[X] host as controller

Reviewer Focus
Describe what reviewers should focus on. Eg.

The event loop could cause the service to exit early if it does not manage object lifetime correctly. Please pay careful attention to how this is achieved in the new event loop.

Future Work
Describe any future work that is required as a result of this change. Eg.

Long-running stress testing needs to be completed.
The old event loop code needs to be removed once stress-testing with libevent has been completed.

Checklist

[ ] Build target all compiles cleanly.
[ ] clang-format and clang-tidy deltas produced no new output.
[ ] Newly added functions include doxygen-style comment block.

I re-targeted this to user/corbinphipps/add-update-config-apis so it's easier to see the changes compared to there since it seems this is a child branch of that.

I re-targeted this to user/corbinphipps/add-update-config-apis so it's easier to see the changes compared to there since it seems this is a child branch of that.

Nevermind, I now re-targeted it such that it can be applied directly to develop without the changes in user/corbinphipps/add-update-config-apis. Limiting the scope should make it easier to review and test. Overall, it probably makes sense to complete this PR first.

SetProtocol sets the hw_mode hostapd property, as well as wmm_enabled and ieee80211*  properties where applicable. Are others needed, such as ht_capab for ieee80211n?

We'll need to figure this out eventually, but we can defer that and do it later on an as-needed, case-by-case basis. I'll create some issues to track that work. The only thing to verify now is whether or not hostapd will refuse to apply the hw_mode setting if the associated settings like ht_capab, vht_capab,  are not specified. As long as it doesn't,

AccessPointControllerTest::SetProtocol currently just returns true. Is this okay to leave unimplemented since this is for a mocked AP?

This one is handled now right, so we can remove this part from the PR description?

In IeeeProtocolToHostapdHwMode, the conversion from ieee80211n to hostapd hw_mode can be either a or g depending on the band, currently it's set to a. How should we handle this? Is there a way to check the bands that are currently set in hostapd somewhere and choose the best option, or should we always set this to a and update it later if a client wants to set a specific frequency band?

This one is difficult because of the hostapd config file implementation. I think what you've done here is good for now, and we can plan to sort out the nuisances later.


Enhancement of the Nav Bar

Hi there,
I was exploring the project and appreciate the work done so far.
I would like to propose the change in the styling of the navbar.
It doesn't seems much engaging and the color after hover doesn't match the background and makes it dull.
I would like to work on this issue and would love to contribute.
Kindly assign me this issue.
Thank you:)

Hi there,
I was exploring the project and appreciate the work done so far.
I would like to propose the change in the styling of the navbar.
It doesn't seems much engaging and the color after hover doesn't match the background and makes it dull.
I would like to work on this issue and would love to contribute.
Kindly assign me this issue.
Thank you:)

Kindly check it @BhuwanSKumar


Basic operator setup

This PR does basic operator setup and mostly carries code structure from CCCMO.

Setup go modules
Add makefile
Add main.go
Setup cluster operator controller
Add cluster api feature gate

/lgtm
/hold
Want to leave others a room to review.

/unhold

/retest


services/gdrive: port code to GdriveCore & add path_2_id cache

Please update with main branch first to avoid possible conflicts.

I will fix them later cause I am going to swim now

I will fix them later cause I am going to swim now

Take your time and have fun!


fsspec source

Is your feature request related to a problem? Please describe.
I haven't yet found a good way to open large (exceeds RAM) remote (not on my local file system) GRIB files in xarray.
Describe the solution you'd like
A new source would be added, e.g.
earthkit.data.from_source(
    "fsspec", uri_or_file, fs=None, **storage_args,
)

that would be similar to the "file" source in making use of random access but use Python's file-like interface (so perhaps "file-like" would be another name) and thus add support for fsspec's numerous backends to earthkit for free.
This new source should also support loading large GRIB datasets without reading the entire file. Ideally, loading the GRIB file into xarray would only read as little data as possible and defer any data reads until the user specifically asks for the data (similar to how NetCDF and Zarr support lazy-loading).
Describe alternatives you've considered
ds = earthkit.data.from_source(
    "stream", fsspec.open("<uri>").open(), batch_size=0,
).to_xarray()

(inspired by https://github.com/ecmwf/cfgrib/issues/326#issuecomment-1604517204) provides the closest current solution but treats the file pessimistically as only a stream and not as a random-access file, which results in excessive reads.
Additional context
I am working in an extremely memory-constrained environment and would like to support opening remote GRIB files (in addition to NetCDF and Zarr datasets which already work).
Organisation
University of Helsinki, ESiWACE3 project

Thank you for the suggestion.
Just a remark. If you want to convert GRIB to xarray first you need to scan the whole file/files (all the messages) for metadata. So this is very much different to use case of NetCDF and zarr where this information is available "instantly".

Thank you for the suggestion.
Just a remark. If you want to convert GRIB to xarray first you need to scan the whole file/files (all the messages) for metadata. So this is very much different to use case of NetCDF and zarr where this information is available "instantly".

I didn’t know that, is it related to GRIB’s format? In that case, would the index files help? If so, would it be possible to check if a pre-generated index file is available as well (e.g. a file-like object passed alongside or a relative fsspec uri) and to use that to skip the initial full-file scan?
In any case, it would be important that once the metadata has been extracted, the actual data is not kept in memory until requested by the user (so slices would still only be lazily loaded in).

I like the idea of fsspec support. I would actually see fsspec as the de-facto standard for data access within the python environment, particularly because widely adopted tools like xarray support this standard natively. I would therefore argue that instead of earthkit developing access to the fsspec drivers, fsspec drivers for fdb, MARS, cds should be developed itself. Those stand-alone drivers would be truely general and can be used across a variety of tools. I just started developing ecmwfspec for this particular reason to provide a general fsspec driver to interact with ECFS (ECMWF File Storage). I could imagine PRs that would extend this to e.g. fdb.


ARC

The project does not support Arc, any chance I can fork the code to make sure it works for ARC?

Sure. Send me a pull request and pull it back. Thanks.
Sent from my iPad
On 5 Feb, 2012, at 4:45 AM<EMAIL_ADDRESS>wrote:

The project does not support Arc, any chance I can fork the code to make sure it works for ARC?

Reply to this email directly or view it on GitHub:
https://github.com/honcheng/iOSPlot/issues/6


Remove associated types from RetroCore trait.

This is a follow-up to #20. The two associated types that were introduced in that PR have been replaced with type parameters with defaults. With this change, users that don't need load_game_special or get_memory are no longer required to explicitly define types for parameters to methods they don't implement.

Sorry for the delay, been getting over a COVID infection. Looking good, I left a few comments.

Thanks! Get well soon. I'll probably have those changes you requested pushed tomorrow, it's getting late today. Happy New Year!

Closing since Adam is no longer available to give final approval on any changes.


feat: Support for Pulse 4.0.0-beta.1

Bumps SPM package version
Fixes tests

Probably not something to merge in until 4.0.0 is final but as I needed to switch to 4.0 and we use this I thought I'd make a branch for myself and others to use in the meantime.

Thanks, @lukeredpath. If you've tried Pulse for Mac or there is something you'd like to see in Pulse 4.0, please let me know!

I have, I’ve submitted a few bugs/enhancement requests via TestFlight. Thanks.

@kean Can this PR be merged? I'm getting a dependency error after updating Pulse 4.0

I pushed version 4.0.0. I appreciate @lukeredpath's work on this repo!


ReferenceError: DEV is not defined

i want test some components ,  but i dont konw how to config the package file , so when i test it, the terminal always  display the error message :ReferenceError: DEV is not defined .

You can add "globals": {"__DEV__": true}" to your config.

What about global function like fbq which is Pixel code. It says ReferenceError: fbq is not defined

Add this code to package.json
{
  "jest": {
   "globals": {
      "__DEV__": true
    }
  }
}


⚠ Bofrost.es has degraded performance

In 49223b5, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1492 ms

Resolved: Bofrost.es performance has improved in 82f2121 after 6 minutes.


SSHLibrary: not possible to change (width/height) of existing session

Robot Framework 3.0.2 (Python 2.7.13 on linux2)
I can set the width with this command and my program can read it accordingly:
Open Connection    host    width=55
But changing the width of the existing connection with this command does not have any effect:
Set Client Configuration    width=222
Although "Get Connections" shows width=222, my program can see width=55. My program can detect windows size changes when using putty or mobaxterm, so I think it is a robot issue.
Also this command works properly:
Set Client Configuration    timeout=5s

Issue moved to SSHLibrary bug tracker:
https://github.com/robotframework/SSHLibrary/issues/248


student's records not there

show, pink slips, tardies, etc

this is fixed in the new version already in the development branch


Browser log, that should be printed inside tests, printed before all tests

Expected behaviour
Every log in browser should be in border of related test
E.g. (https://github.com/ilgonmic/karma-ts-test )
HeadlessChrome 77.0.3844 (Mac OS X 10.14.6)  MyTest foo FAILED
        Error: foo
            at MyTest.foo (test.js:25:9)
            at Context.<anonymous> (test.js:31:20)

// for passed test
ERROR: 'MY FAIL'
LOG: 'MY LOG
LOG ON NEXT LINE'

//for failed test
ERROR: 'MY FAIL'
LOG: 'MY LOG
LOG ON NEXT LINE'
HeadlessChrome 77.0.3844 (Mac OS X 10.14.6) org.my ServiceTest valueFailTest FAILED
        Error: valueFailTest
            at ServiceTest.valueFailTest (test.js:16:51)
            at Context.<anonymous> (test.js:43:25)

HeadlessChrome 77.0.3844 (Mac OS X 10.14.6) org.my ServiceTest valueFailTest2 FAILED
        Error: valueFailTest2
            at ServiceTest.valueFailTest2 (test.js:20:51)
            at Context.<anonymous> (test.js:47:25)
HeadlessChrome 77.0.3844 (Mac OS X 10.14.6): Executed 4 of 4 (3 FAILED) (0.006 secs / 0 secs)

Actual behaviour
All logs printed before test reporting
ERROR: 'MY FAIL'
LOG: 'MY LOG
LOG ON NEXT LINE'
ERROR: 'MY FAIL'
LOG: 'MY LOG
LOG ON NEXT LINE'
ERROR: 'MY FAIL'
LOG: 'MY LOG
LOG ON NEXT LINE'
HeadlessChrome 77.0.3844 (Mac OS X 10.14.6)  MyTest foo FAILED
        Error: foo
            at MyTest.foo (test.js:25:9)
            at Context.<anonymous> (test.js:31:20)
HeadlessChrome 77.0.3844 (Mac OS X 10.14.6) org.my ServiceTest valueFailTest FAILED
        Error: valueFailTest
            at ServiceTest.valueFailTest (test.js:16:51)
            at Context.<anonymous> (test.js:43:25)
HeadlessChrome 77.0.3844 (Mac OS X 10.14.6) org.my ServiceTest valueFailTest2 FAILED
        Error: valueFailTest2
            at ServiceTest.valueFailTest2 (test.js:20:51)
            at Context.<anonymous> (test.js:47:25)
HeadlessChrome 77.0.3844 (Mac OS X 10.14.6): Executed 4 of 4 (3 FAILED) (0.006 secs / 0 secs)
TOTAL: 3 FAILED, 1 SUCCESS

Environment Details

Karma version (output of karma --version):
Karma version: 4.3.0
Relevant part of your karma.config.js file

  config.set({
    "singleRun": true,
    "autoWatch": false,
    "files": [
      "/Users/ilya.goncharov/repos/karma-ts-test/test.js"
    ],
    "frameworks": [
      "mocha"
    ],
    "browsers": [
      "ChromeHeadless"
    ]
  });

Steps to reproduce the behaviour

Small sample (https://github.com/ilgonmic/karma-ts-test) with only karma and mocha framework
Test with console.log in body
node_modules/.bin/karma start

Is there known problem? And is there some workaround for it?

The test logs are printed as they happen, during the test.  The test results are printed by the reporter you select.  So to have them integrated you need to change/rewrite the reporter.

@johnjbarton So, it is expected behaviour for default 'progress' reporter, am I right?

Yes that's my experience. Unfortunately all of the reporters I've seen seem to follow that pattern.

Just for history
It can be solved via window.__karma__.result property
It should be patched in 'client' side (via framework or directly in test code) like this
const resultFn = window.__karma__.result
window.__karma__.result = function(result) {
    console.log(`TEST_ENDED\n${JSON.stringify(result)}`
    resultFn(result)
}

And then it should be catched on reporter side vid onBrowserLog message
In reporter, we can collect information of logs and associate it with tests


fix #238 replace package object by all object

you cannot deprecate a package object so this change is not binary compatible

@aoiroaoino Thanks for your careful review. Turns out I had forgotten to delete the package in mocle.std that's why Stream` import in was not required.
I had a go with -Ywarn-unused-import but I got some nonsensical error, so I removed it. Maybe I did something wrong, we can always add it in a later PR if someone can make it work correctly :)
I added all traits to MonocleSuite hence we don't need so many imports in tests.

@aoiroaoino do you think it is correct now?

sorry, for the late reply...
I have checked it. And it's correct.
LGTM :+1:

Thanks for the review!


EPERM, unlink Microsoft.CodeAnalysis.CSharp.dll when updating

On Windows, every update I perform fails with an EPERM error releated to unlinking some library or other.
Installing<EMAIL_ADDRESS>to C:\Users\afontaine\.atom\packages failed
EPERM, unlink 'C:\Users\afontaine\.atom\packages\omnisharp-atom\node_modules\omnisharp-client\roslyn\packages\dnx-clr-win-x<IP_ADDRESS>-beta4\bin\Microsoft.CodeAnalysis.CSharp.dll'

Do I need admin privileges to update omnisharp-atom on Windows?

It's a known issue with the Windows updater. It can't replace the executables while in use.
For now, close Atom, then apm install omnisharp-atom from the command line.

Ah understandable. Thanks!


Stop download Red Hat manifests from the image

Once https://github.com/enterprise-contract/ec-policies/issues/1115 is completed and enough time has passed for users to adopt it, the EC cli no longer needs to download the Red Hat manifests from the image. The rego will do it if needed.
Acceptance Criteria

EC cli no longer downloads Red Hat manifests from within the image.

Fixed via https://github.com/enterprise-contract/ec-cli/pull/1986


BUG: stats: underflow in geom distribution

Describe your issue.
stats.geom.rvs returns negative values when p is small.  The support is strictly positive and should never return these values.
When validating this in R, I am able to get valid values for substantially smaller values of p and this should mirror that kind of performance.
Reproducing Code Example
from scipy.stats import geom

# This returns a valid output
geom.rvs(0.0000000001, random_state = 1234) #(2125986401)
# Adding an extra zero will cause it to fail as will any smaller number
geom.rvs(0.0000000001, random_state = 1234) #(-2147483648)

Error message
-2147483648

SciPy/NumPy/Python version and system information
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.10.1 1.23.5 sys.version_info(major=3, minor=10, micro=9, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: c:/opt/openblas/if_32/64/include
    lib directory: c:/opt/openblas/if_32/64/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= PRESCOTT MAX_THREADS=4
    pc file directory: c:/opt/openblas/if_32/64/lib/pkgconfig
    version: 0.3.18
  lapack:
    detection method: pkgconfig
    found: true
    include directory: c:/opt/openblas/if_32/64/include
    lib directory: c:/opt/openblas/if_32/64/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= PRESCOTT MAX_THREADS=4
    pc file directory: c:/opt/openblas/if_32/64/lib/pkgconfig
    version: 0.3.18
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 0.29.33
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  pythran:
    include directory: C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-v_zt9njv\overlay\Lib\site-packages\pythran
    version: 0.12.1
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
Python Information:
  path: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-dskf9r9k\cp310-win_amd64\build\venv\Scripts\python.exe
  version: '3.10'

Hi @eluerken-wm, thank you for reporting.
This is actually an error coming from NumPy. Behind the scene, stats.geom.rvs is making a call to the geometric distribution from NumPy.
np.random.geometric(p=1e-20)
# -9223372036854775808

Now, this issue is only present with the legacy random number generator interface. And if you use the new np.random.Generator, the problem is fixed:
np.random.default_rng().geometric(p=1e-20)
#<PHONE_NUMBER>854775807

And you can verify this in SciPy by passing a proper Generator object, which is the recommended way:
geom.rvs(1e-20, random_state=np.random.default_rng())
#<PHONE_NUMBER>854775807

Hence, I am going to close this issue.

I cannot reproduce what you are getting:
import scipy
np.__version__, scipy.__version__
# ('1.24.3', '1.11.0.dev0+1967.039e931')
from scipy.stats import geom
rng = np.random.default_rng()
rng.geometric(1e-20, size=15)

array([9223372036854775807,<PHONE_NUMBER>854775807,<PHONE_NUMBER>854775807,
<PHONE_NUMBER>854775807,<PHONE_NUMBER>854775807,<PHONE_NUMBER>854775807,
<PHONE_NUMBER>854775807,<PHONE_NUMBER>854775807,<PHONE_NUMBER>854775807,
<PHONE_NUMBER>854775807,<PHONE_NUMBER>854775807,<PHONE_NUMBER>042018048,
<PHONE_NUMBER>854775807,<PHONE_NUMBER>854775807,<PHONE_NUMBER>854775807])

Maybe there is more to it then: I am on a M1, could be windows things. It seemed to be linked to NumPy and the use of something else than Generator. That was my assumption for making the call.
Feel free to reopen and propose a workaround in SciPy if you feel otherwise.

It's certainly exacerbated by Windows and RandomState, but not exclusive to it. On 64-bit Linux (Intel), we also get negatives.
>>> import numpy as np

>>> rng = np.random.default_rng(0x30627150c0e97f971ae8da6864ceb85c)

>>> rng.geometric(1e-20, size=15)
array([-9223372036854775808, -9223372036854775808, -9223372036854775808,
       -9223372036854775808, -9223372036854775808,<PHONE_NUMBER>578648064,
       -9223372036854775808, -9223372036854775808, -9223372036854775808,
       -9223372036854775808, -9223372036854775808, -9223372036854775808,
       -9223372036854775808, -9223372036854775808, -9223372036854775808])

>>> np.__version__
'1.25.0.dev0+1126.g4268399a9'

It might be some different behavior in what (int64_t)ceil(...) does on the different compilers. While there is probably a reasonable upstream fix (forcing the result in such cases to be np.iinfo(np.int64).max), it can still be worth doing the workaround in geom.rvs() as well in both RandomState and Generator cases.

It might be some different behavior in what (int64_t)ceil(...) does on the different compilers.

Yes, that is the problem; see my recent comment in the numpy issue.

If I understand correctly, the proposed patch would change the results in this case from negative numbers, which are obviously not geometrically distributed, to positive numbers that are also not geometrically distributed, but less obviously so. If that's correct, I'd prefer not to implement this.

Thank you so much @tupui, @WarrenWeckesser  & @rkern!  I am sorry that I didn't reply sooner, I didn't get these notifications.  This was super informative.
I'm trying to go through the threads here, but it still seems like this issue is not fully resolved, is that fair?  I am still getting the same issues and it sounds like the proposed solution is to just flip the negative sign, is that correct?

We never actually give out geometrically distributed numbers because of the limitations of finite floating point and integer arithmetic. We always have a cutoff. As a matter of design of practical numerical distribution samplers, I'm very happy to have all of the probability mass in the excluded tail be assigned to the largest representable integer. This is the case even when the parameters are of typical size and only negligible amounts of probability mass are in that excluded tail; this case with extreme parameters isn't really exceptional. The overflow to negatives is a pure accident of (my) careless implementation and not a helpful feature in any way. We've already made the change for Generator; I think we should fix up the results that come from RandomState and older versions of Generator here.

I am still getting the same issues and it sounds like the proposed solution is to just flip the negative sign, is that correct?

Not quite: make the result np.iinfo(result.dtype).max. That's typically one less than the negative of the negative integer that you get.

@rkern it looks like you reopened this. Are you willing to submit a fix or review one? If so, I'm willing to help find someone to do the other half so that we can resolve this.

I'll certainly review.

@OmarManzoor @dschmitz89 I thought I'd bring this issue to your attention in case you'd like to submit a PR.

From what I understood @rkern 's proposed solution is to swap out all negative values for the maximum possible integer.
Would it not be better to raise an error or at least a RuntimeWarning? Otherwise, the method will silently fail without users noticing it.

The behavior in this edge cases used to be undefined behavior in C. One some compilers and platforms, you got the maximum integer and others you got the most negative integer. We've fixed it upstream in Generator to uniformly do the former. The remainder to fix here is for the codepaths still using the legacy RandomState which will still be platform-dependent.
I'm definitely against raising an error dependent on the values that come out the end; that's just a recipe for heisenbugs that are difficult to track down. I'm mildly okay with warnings based on the input parameters where the expectation of drawing from the excluded region hits some threshold. I'm not in any particular mood to define that threshold, though.
A reminder: all of the results from our finite implementations are "failures" to draw from the true distribution. Some are just more obvious than most.

I'm mildly okay with warnings based on the input parameters where the expectation of drawing from the excluded region hits some threshold. I'm not in any particular mood to define that threshold, though.

TLDR: For p > 1e-17, it's very unlikely to ever get observations clipped to the maximum int64; for p < 1e-20, it's all but guaranteed. So I'd suggest warning if p < 1e-17, assuming that it is relatively rare for people to be sampling with 1e-20 < p < 1e-17 and get the warning unnecessarily.
import numpy as np
max_int = np.iinfo(np.int64).max
rng = np.random.default_rng()
((rng.geometric(1e-17, size=10000000) >= max_int).any(),  # False
 (rng.geometric(1e-20, size=1) >= max_int).any())  # True

Given the number of observations requested, the maximum integer, and a threshold probability of getting an observation from the excluded region, the algebra gives a simple expression for the corresponding threshold for the shape `p`.
x = 1000  # maximum int
n = 100  # number of observations from geometric distribution
pn = 0.75  # probability of at least one observation > x
p1 = 1 - (1 - pn)**(1/n)
p = 1 - p1**(1/(x + 1))
# approximation when p is very small: abs(1/(x + 1) * np.log(p1))

import numpy as np
rng = np.random.default_rng(398234982523)
any_gt_x = []
for i in range(10000):
    any_gt_x.append((rng.geometric(p, size=n) > x).any())
np.mean(any_gt_x)  # 0.7519

For example, if x is the maximum int64, 1 million observations are drawn, and our threshold probability is 0.1%, the threshold p is 2.4851459794772703e-18.
So if p < 2.485e-18, there is at least a 0.1% chance of one or more of the one million observations being clipped at the maximum integer.
But if we set x the maximum int64, the threshold for p is surprisingly insensitive to the number of observations (up to say, 1e15) and even the probability threshold we choosem, leading to the TLDR suggestion.

I think it’s better to handle the situation based on the output rather than input. After all, for any input there is non-zero probability that the generated random number overflows; the function still has to handle this anyway. And warning on small input seems too eager to me: it’s like warning on allocating a huge block of memory that has 99% chance of failure and 1% chance of success; I think the typical behavior is to just do it and report failure if so.
While floating arithmetic is approximate and is usually addressed by rounding, the situation of integer overflow here is a bit different. Saturation is not rounding, and truncation is not approximation. Therefore I think it’s worth doing something in addition to saturation (which I think is a reasonable choice for the integer domain).
If a RuntimeWarning is deemed too noisy, I think it is worth at least to document the saturation behavior, which seems to me not the most obvious choice. (To me the more “Pythonic” behavior might be returning a “BigInteger” or raising a RuntimeError.)

I definitely agree that the saturation behavior should be documented. I still think it's acceptable to do nothing except document the behavior (Matt's findings concerning the parameter space is very informative to include, as well as an example of how one can check for saturation oneself).
But if we do want to have a warning, let me make one more pitch for doing it based on the passed input parameters rather than observing a saturated output. My main principle when it comes to warnings is that they need to be actionable for the user of the function. If the user can't change anything under their control to avoid triggering the warning, don't warn. As a kind of corollary, I think that when users make choices under their control that would lead to the warning-worthy outcomes depending on the things that are outside of their control, we should usually still warn them. Otherwise, we get heisenbugs that are hard to track down. A coder uses geom.rvs() with these extreme parameters and it happens to work fine with the PRNG state they tested it, but then in actual use, there are a bunch of warnings. But only sometimes.
There are two things that determine whether or not this rvs() method saturates the output: the parameters and the PRNG state. Practically speaking, the user only controls the parameters, not the PRNG state (certainly not in a way to control whether or not it saturates, except in contrived, party-trick scenarios). Every PRNG state is equally valid (or invalid), so I think it behooves us to think in terms of the ensemble of possible results when deciding to warn, not just when a particular PRNG state happens to push the result to the saturation limit. Our finite arithmetic always distorts the probability distribution, even aside from saturation. Under this rubric, we consider it warning-worthy when the distortion is high enough (for some value of "enough").  Even when we have very reasonable p values like 0.3, we're distorting the distribution. AFAICT, when p==0.3, our algorithm can never generate the maximum int64 integer (or even much smaller integers); it is not actually true that every parameter choice has a finite chance of saturating. We just don't care about that particular distortion because the true probabilities were so low to begin with that the amount of distortion to the distribution is small. Working through the ziggurat constants, I think it's really only possible to start saturating at around p=1e-18.
Also, more specific to this case, when p<eps, in addition to the saturation, we also start to observe wider and wider gaps between possible integer outputs below the saturation limit. That's an additional distortion that is only possible to detect by looking at the input parameters and not the result.

My main principle when it comes to warnings is that they need to be actionable for the user of the function. If the user can't change anything under their control to avoid triggering the warning, don't warn. As a kind of corollary, I think that when users make choices under their control that would lead to the warning-worthy outcomes depending on the things that are outside of their control, we should usually still warn them. Otherwise, we get heisenbugs that are hard to track down. A coder uses geom.rvs() with these extreme parameters and it happens to work fine with the PRNG state they tested it, but then in actual use, there are a bunch of warnings. But only sometimes.

This is very convincing to me!

What should happen when the function can detect that there may be a problem but:
a) can't be sure, and
b) can't suggest what to do (besides solving a different problem).
For example:
import numpy as np
from scipy import linalg
rng = np.random.default_rng()
n = 1000
A = np.tril(rng.random((n, n)))
b = rng.random(n)
x = linalg.solve(A, b)
# LinAlgWarning: Ill-conditioned matrix (rcond=1.55382e-21): result may not be accurate.

What should this do?

I'm not necessarily saying that the warning message has to suggest any particular action, just that whether we issue a warning or not shouldn't hinge on something that's outside of a user's control. We often don't have nearly enough information about the user's context to suggest the right place to intervene, which might be far upstream of our library function. People solve linear systems for lots of reasons in lots of different domains, and how the ill-conditioned matrix comes into play varies. The root cause might be because the mesh in the PDE discretization is less than ideal. It might be something that could be resolved by numerical preconditioning (possibly with a different kind of solver) regardless of where the matrix comes from.
No warning message has to be comprehensive, in fact, being short and internet-searchable can be more valuable because people can find documentation and discussions that help them explore their options with higher context.

@rkern Makes sense to me!
While the shape parameter has to be really small to trigger saturation (with non-negligible probability) for the geometric distribution, saturation could be more frequent for fat-tailed distributions. For example, for the Pareto distribution (whose survival probability is $x^{-\alpha}$), if $\alpha=0.001$, there is about half chance that a random number saturates to np.inf.
The saturation behavior for floating-point numbers could be used as a reference for integers. For most operations defined in IEEE-754, in addition to returning +inf, a (hardware) floating point exception is often raised unless it is suppressed (but it is often suppressed in practice).
There is no inf for integers, so the max integer is a natural choice for saturation.
Raising an error (or warning) on integer saturation might be a bit too intrusive, especially considering that floating point exceptions are often suppressed for this reason. Therefore I agree that documenting the saturation behavior sounds good enough (and doing more might not be desirable).
(As for the Pareto distribution, since it is known as an extreme value distribution and is often used to model excess over a threshold, the saturation behavior is particularly appropriate for such usage.)

The underflow issue is resolved, so closing. Follow-up requests for warnings may be opened with NumPy, and SciPy will follow suit.


R-T5.4-5: The data pipeline module of the Orchestrator should support deployment of data pipelines which automate movement of data between two or more clouds

ID:
R-T5.4-5

Type:
FUNCTIONAL_SUITABILITY

User   story:
As an Operations Engineer, I want to be   able to deploy data pipelines across multiple (cloud) systems

Requirement:
The data pipeline module of the Orchestrator should support deployment of data pipelines which automate   movement of data between two or more clouds

Extended   Description:
Goal is to support data pipelines that   migrate data between different cloud providers and availability zones.

Priority:
Should have

Affected   Tools:
ORCHESTRATOR

Means   of Verification:
Test data pipeline orchestration where   different parts of the pipeline are deployed on different clouds

Take a look at for my first comment here: #17.

A number of TOSCA node types are created to support the movement of data between two or more clouds.
ConsGCSBucket, ConsAzureBlob, and ConsS3Bucket are developed to consume data from
Google Cloud Storage, Azure blob storage, and S3 bucket, respectively.
Similarly, PubGCS, PubsAzureBlob and PubsS3Bucket are developed to publish data to Google Cloud Storage, Azure blob storage, and S3 bucket, respectively.
For private cloud, we have developed ConsumeLocal and PublishLocal.
This issue can be closed now. @gcasale

Approved


How to use joinedload options when using sqlalchemy

Hello!
I am trying to use joinedload with sqlalchemy but it seems overwriting original query because of passing by reference.
https://github.com/uriyyo/fastapi-pagination/blob/0ce8f30e321caece96a2ba6c0261f9a230d78ec2/fastapi_pagination/ext/sqlalchemy.py#L55
Are there any way to use options when using sqlalchemy?

Hi @fshmng09,
It actually doesn't override your original query. sqlalchemy creates a new query object when you call most of the query methods:
assert query is not query.order_by(None)

@uriyyo
Hi! Thank you for you reply!
I have created a code to reproduce the problem.
https://github.com/fshmng09/fastapi-pagination-postgres-sample
In this sample, user-todo has a 1-to-N relationship.
When the list of users is GETted, the todo associated with the user is printed.
Also, when the query is executed, the sql statement is printed.
We consider three patterns:

GET:/users
GET:/joined-users
GET:/paged-joined-users.
Each log is stored under logs/.

When executing GET:/users, the query is executed multiple times.
In the case of GET:/joined-users , the query is executed only once at first.
In GET:/paged-joined-users, paging is added and printed. In fact, the query is executed multiple times, just as it would be if it were not joinedloaded.
This is a major performance concern for complex table.
I would like to solve this problem.

https://github.com/fshmng09/fastapi-pagination-postgres-sample/blob/dea5c672dccea9ca3fcaec2f0ee11a29428d3ddb/app/endpoints.py#L69
Please, replace replace joinedload with selectinload:
from sqlalchemy.orm import selectinload

@api_router.get("/paged-joined-user", response_model=Page[schema.User])
def get_paged_joined_users(db: Session = Depends(deps.get_db), params: Params = Depends()) -> Page[schema.User]:
    paged_users = paginate(db.query(models.User).options(
        selectinload(models.User.todos)
    ), params)
    print_for_check_query(paged_users.items)
    return paged_users

You will have 3 queries being executed:

To get total count of rows.
To load rows chunk.
To load relationships.

@uriyyo
Thank you for your checking!!
I made a mistake with get_joined_users.txt in the previous repository. My sincere apologies.
I also tried selectinload as you suggested, but it seems that todoos are selected by the number of todos. subqueryload is the same. (total of four queries are executed, two of which refer to the todo table.)
I pushed the above two fixes to the sample repository.

@uriyyo
Hello.
When I tried items=query.all() here, the joinedload is done as expected and the N+1 problem is solved.
https://github.com/uriyyo/fastapi-pagination/blob/0a06e664e13825c6df2be8acb962180fe4bc533a/fastapi_pagination/ext/sqlalchemy.py#L115

@fshmng09 Hmm, that's interesting, I will take a look

@fshmng09 New version 0.12.5 has been released. This issue should be fixed now.

@uriyyo
Thank you so much.
It is working well now!!

Great, happy to hear that


Convert beluga_ros from header-only to a compiled library

Proposed changes
Related to #385.
Improves memory usage during compilation of the classic amcl node.
Type of change

[ ] 🐛 Bugfix (change which fixes an issue)
[x] 🚀 Feature (change which adds functionality)
[ ] 📚 Documentation (change which fixes or extends documentation)

Checklist

[x] Lint and unit tests (if any) pass locally with my changes
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have added necessary documentation (if appropriate)
[x] All commits have been signed for DCO

Additional comments
The total memory usage does not improve when compiling the whole beluga_amcl package as the NDT nodes are the bottleneck now. I will work on that in follow-up PRs.

Results after commenting out NDT node targets in beluga_amcl.
colcon build --packages-up-to beluga_amcl --event-handlers console_direct+ --mixin build-testing-off

Before: Maximum resident set size (kbytes): 2045884
After: Maximum resident set size (kbytes): 1649624

Nice!


Add clickbench queries to tpch and bench.sh

Is your feature request related to a problem or challenge?
The TPCH queries, while good, also have joins and other things that make performance comparisons of grouping performance challening
The ClickBench benchmarks are both widely used in the industry and focus on grouping / aggregation / filtering: https://github.com/ClickHouse/ClickBench/tree/main/datafusion
Describe the solution you'd like
Add clickbench queries to tpch benchmark runner and bench.sh
We can adapt the code in
https://github.com/ClickHouse/ClickBench/tree/main/datafusion
Describe alternatives you've considered
No response
Additional context
No response

I am  working on https://github.com/apache/arrow-datafusion/issues/7052


WEBAPP-643: Fix display sub categories

This pull request belongs to an issue on our bugtracker.
You can find it there by looking for an issue with the key which is mentioned in the title of this pull request.
It starts with the keyword WEBAPP.

@maxammann You can merge this, once you merge WEBAPP-640... My reaction time is probably quite high these days :see_no_evil:


Error updating vscode - Windows 10

VSCode Version: 1.43.2
OS Version: Windows 10

Steps to Reproduce:
On windows vscode promots update now and when trying to update it craps out with an error.
I've reinstalled vscode completely and still face this.

Does this issue occur when all extensions are disabled?: Yes/No

VSCode Version: 1.44
OS Version: Windows 10 Pro

I just had this problem. I had to close malwarebytes and rerun a downloaded installer, then it worked.

It seems malwarebytes "locks" %appdata% and makes the application look like its not installed anymore

Ouch! My laptop is connected to my work domain and I don't think I can
disable the Malwarebytes or any other antivirus software. Any workarounds?
On Tue, Apr 14, 2020, 2:23 PM 0xicl33n<EMAIL_ADDRESS>wrote:

VSCode Version: 1.44
OS Version: Windows 10 Pro

I just had this problem. I had to close malwarebytes and rerun a
downloaded installer, then it worked.
[image: image]
https://user-images.githubusercontent.com/2192160/79259682-f369e980-7e41-11ea-9605-77d888f13b26.png
It seems malwarebytes "locks" %appdata% and makes the application look
like its not installed anymore
[image: image]
https://user-images.githubusercontent.com/2192160/79259804-257b4b80-7e42-11ea-8447-b099bdb6550a.png
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode/issues/95257#issuecomment-613603388,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACYIUQYCRN4E6X3UBDTMKPTRMSSYJANCNFSM4MH6EISQ
.

/duplicate #47841
As a workaround, please disable Windows background updates by setting update.enableWindowsBackgroundUpdates to false.


Support for AlloyDB?

Hi,
It's not a bug but had a question and did not find any link for raising a question. Does this library also support AlloyDB which is an extension of Postgre-sql? I could not find it in the documentation so, wanted to confirm

We use python postgresdb writer to write to Postgres. We have not tested it
on Alloydb.
On Tue, Dec 5, 2023 at 2:18 AM Sumit Desai @.***> wrote:

Hi,
It's not a bug but had a question and did not find any link for raising a
question. Does this library also support AlloyDB which is an extension of
Postgre-sql? I could not find it in the documentation so, wanted to confirm
—
Reply to this email directly, view it on GitHub
https://github.com/pysql-beam/pysql-beam/issues/12, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAQVCMTTQYF4DNBJ2R2YYODYH3KFLAVCNFSM6AAAAABAHIH3AOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZDKNJWGQ4DGNA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

All right, thanks. Will try it out with AlloyDB

I am trying to use the latest version in my pyproject.toml. However, I am getting following error-
Because pysql-beam (0.2) depends on psycopg2-binary (2.8.2). and no versions of pysql-beam match >0.2,<0.3, pysql-beam (>=0.2,<0.3) requires 				psycopg2-binary (2.8.2). So, because up-trend-analyzer-service depends on both psycopg2-binary (^2.9.9) and pysql-beam (~0.2), version solving failed.
psycopg2-binary is more than 4 years old. Is there any reason why pysql-beam is using 4 years old version of psycopg2-binary?

Yes, I was worried about similar errors. It was developed few years ago due
to my current work policies note able to make it current. I will try to
test it in coming days to work with latest version . Contributions are very
much needed for this tool to grow.
On Tue, Dec 5, 2023 at 11:51 PM Sumit Desai @.***>
wrote:

I am trying to use the latest version in my pyproject.toml. However, I am
getting following error-
Because pysql-beam (0.2) depends on psycopg2-binary (2.8.2). and no
versions of pysql-beam match >0.2,<0.3, pysql-beam (>=0.2,<0.3) requires
psycopg2-binary (2.8.2). So, because up-trend-analyzer-service depends on
both psycopg2-binary (^2.9.9) and pysql-beam (~0.2), version solving failed.
psycopg2-binary is more than 4 years old. Is there any reason why
pysql-beam is using 4 years old version of psycopg2-binary?
—
Reply to this email directly, view it on GitHub
https://github.com/pysql-beam/pysql-beam/issues/12#issuecomment-1842132815,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAQVCMWH5KGW5E4F2QIGX7TYIABW3AVCNFSM6AAAAABAHIH3AOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBSGEZTEOBRGU
.
You are receiving this because you commented.Message ID:
@.***>


Support humble CI

・Change CI from foxy to humble

https://answers.ros.org/question/406709/sudden-custom_msgs-related-error-in-ros2-build-failed-in-ci/


Add canary region to smoke test matrix

This PR adds a new entry to the smoke test matrix to add a public cloud test against a Canary/EUAP region (specifically eastus2euap).
Azure/azure-sdk#1879

/azp run java - smoke-tests

/azp run java - smoke-tests

/check-enforcer reset

/check-enforcer evaluate

/check-enforcer override

Pipeline seems really flaky with timeouts, but I have a passing run at https://dev.azure.com/azure-sdk/internal/_build/results?buildId=572825&view=results


[BugFix] fix orc pushdown filter failure

What type of PR is this：
BugFix
Problem Summary(Required) ：
When ORC executes filter pushdown, assuming there are two predicates in the SQL query, when the function filterOnPickStringDictionary is executed, if predicate1 is present in sdicts, dictionary filtering can be performed and dict_value_chunk->num_rows() after filtering is greater than 0, then _dict_filter_eval_cache will not be cleared. However, if predicate2 is not present in sdicts, the dictionary filtering in filterOnPickStringDictionary will not be executed, and eval_conjuncts_into_filter will obtain chunk_size of 0, which will execute the next chunk retrieval operation. It can be seen that _dict_filter_eval_cache will not be properly cleared. When the next batch of stripe data is retrieved and _orc_row_reader_filter->is_slot_evaluated(it.first) returns true, it will skip verification even if it does not satisfy the predicate."

@Smith-Cruise

Thanks @kobebryantlin0 . It's quite hard to catch that.
Could you be kind to give an exmaple/ut to reproduce this case?

Thanks @kobebryantlin0 . It's quite hard to catch that.
Could you be kind to give an exmaple/ut to reproduce this case?

Example
SELECT id, name FROM table WHERE dt >= CAST("2023-03-30" AS string) AND dt <= CAST("2023-03-30" AS string) AND id = '003' AND outline_1 = '投诉' limit 1;
+-----------------+
| id.   | name |
+-----------------+
| 001 | 咨询 |
| 002 | 投诉 |
| 003 | 反馈 |
| 004 | 咨询 |
+-----------------+-
Real Output:
+-----------------+
| id | name |
+-----------------+
| 003 | 反馈 |
Expected Output:
empty set
In the sample data above, 001 and 002 belong to the first batch of chunks, while 003 and 004 belong to the second batch of chunks. In the first batch of two data rows, the id field returns false for filterOnPickStringDictionary because sdicts is empty, so data filtering is not performed. For the 002 record, the name field matches, so _dict_filter_eval_cache is not cleared. Moving on to the second batch of chunk data, the id field matches for the 003 record, but the name field is not validated because the _dict_filter_eval_cache from the previous batch was not cleared, so data filtering does not take effect in this case.

please add ut

Does it mean t

Thanks @kobebryantlin0 . It's quite hard to catch that.
Could you be kind to give an exmaple/ut to reproduce this case?

Example
Raw Data +-----------------+ | id. | name | +-----------------+ | 001 | 咨询 | | 002 | 投诉 |
| 003 | 反馈 | | 004 | 咨询 | +-----------------+-
Query Sql
SELECT id, name FROM table WHERE dt >= CAST("2023-03-30" AS string) AND dt <= CAST("2023-03-30" AS string) AND id = '003' AND name = '投诉' limit 1;
Real Output:
+-----------------+ | id | name | +-----------------+
| 003 | 反馈 |
Expected Output:
empty set
In the sample data above, 001 and 002 belong to the first batch of chunks, while 003 and 004 belong to the second batch of chunks. In the first batch of two data rows, the id field returns false for filterOnPickStringDictionary because sdicts is empty, so data filtering is not performed. For the 002 record, the name field matches, so _dict_filter_eval_cache is not cleared. Moving on to the second batch of chunk data, the id field matches for the 003 record, but the name field is not validated because the _dict_filter_eval_cache from the previous batch was not cleared, so data filtering does not take effect in this case.
@Smith-Cruise @dirtysalt

Good catch!
In this example, do you mean there are two stripe: stripe one contains two row: | 001 | 咨询 | | 002 | 投诉 |
stripe two contains other two row: | 003 | 反馈 | | 004 | 咨询 |
For the first stripe, there exists sdicts, so the _dict_filter_eval_cache is evaluated. but for the second stripe, there are no sdicts, so the _dict_filter_eval_cache is not evaluated and not cleaned. later we use the wrong _dict_filter_eval_cache to filter data
as you can see, if sdicts is empty, it means StringVectorBatch::use_codes will be false too, so there is another issue is in OrcChunkReader::apply_dict_filter_eval_cache we don't check if StringVectorBatch::use_codes is false or not to decide whether to filter it by _dict_filter_eval_cache

@DorianZheng
You are right, there was probably an issue before, but with the changes I made in my submitted PR, I believe we do not need to check for it in apply_dict_filter_eval_cache anymore. This is because after my modifications, filterOnPickStringDictionary always clear dict_filter_eval_cache before returning, regardless of whether sdicts is empty or not. Since filterOnPickStringDictionary is executed before apply_dict_filter_eval_cache, theoretically dict_filter_eval_cache should have already been cleared in the former if use_codes is false. Additionally, apply_dict_filter_eval_cache has a check for whether dict_filter_eval_cache.size() is 0, so subsequent logic should not be executed.

@kobebryantlin0 Thanks for the fix. Could you add UT.

To normal orc writer, there will never be the case that two stripes use different dictionary encoding schema.  Because there is a configuration item said that
  ROW_INDEX_STRIDE_DICTIONARY_CHECK("orc.dictionary.early.check",
      "hive.orc.row.index.stride.dictionary.check",
      true,
      "If enabled dictionary check will happen after first row index stride\n" +
          "(default 10000 rows) else dictionary check will happen before\n" +
          "writing first stripe. In both cases, the decision to use\n" +
          "dictionary or not will be retained thereafter."),

So to reproduce this case, I have to make some modification on java orc code like following:
diff --git a/java/core/src/java/org/apache/orc/impl/writer/StringBaseTreeWriter.java b/java/core/src/java/org/apache/orc/impl/writer/StringBaseTreeWriter.java
index 9d87873e7..d320df7b4 100644
--- a/java/core/src/java/org/apache/orc/impl/writer/StringBaseTreeWriter.java
+++ b/java/core/src/java/org/apache/orc/impl/writer/StringBaseTreeWriter.java
@@ -139,6 +139,7 @@ public abstract class StringBaseTreeWriter extends TreeWriterBase {
     // reset all of the fields to be ready for the next stripe.
     dictionary.clear();
     savedRowIndex.clear();
+    doneDictionaryCheck = false;
     rowIndexValueCount.clear();
     if (rowIndexPosition != null) {
       recordPosition(rowIndexPosition);

And by generating following file and table schema, I can reproduce this case
package org.apache.orc.examples;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector;
import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
import org.apache.orc.OrcFile;
import org.apache.orc.TypeDescription;
import org.apache.orc.Writer;

import java.io.File;
import java.util.Random;

public class GenORCFileExample {
    public static String genRandomString(int size) {
        Random rnd = new Random();
        StringBuffer sb = new StringBuffer();
        for (int i = 0; i < size; i++) {
            char ch = (char) (rnd.nextInt(100) % 26 + 'a');
            sb.append(ch);
        }
        return sb.toString();
    }

    public static void genTwoStripsDictAndNoDict() throws Exception {
        Configuration conf = new Configuration();
        int batchSize = 100;
        conf.setDouble("orc.dictionary.key.threshold", 0.01);
        conf.setLong("orc.stripe.size", 10);
        conf.setLong("orc.stripe.row.count", batchSize);

        TypeDescription schema = TypeDescription.fromString("struct<x:int,y:string>");
        Path p = new Path("/Users/dirlt/Downloads/two-strips-dict-and-nodict.orc");
        File f = new File(p.toString());
        if (f.exists()) {
            f.delete();
        }
        Writer writer = OrcFile.createWriter(p,
                OrcFile.writerOptions(conf)
                        .setSchema(schema));

        VectorizedRowBatch batch = schema.createRowBatch(batchSize);
        LongColumnVector x = (LongColumnVector) batch.cols[0];
        BytesColumnVector y = (BytesColumnVector) batch.cols[1];
        for (int r = 0; r < batchSize; ++r) {
            x.vector[r] = r % 10;
            y.setVal(r, "a".getBytes());
        }
        batch.size = batchSize;
        writer.addRowBatch(batch);
        batch.reset();

        x.vector[0] = 1;
        y.setVal(0, "a".getBytes());
        for (int r = 1; r < batchSize; ++r) {
            x.vector[r] = r % 10;
            y.setVal(r, genRandomString(20).getBytes());
        }
        batch.size = batchSize;
        writer.addRowBatch(batch);
        batch.reset();

        writer.close();
    }

    public static void main(String[] args) {
        try {
            genTwoStripsDictAndNoDict();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

}

table schema
CREATE TABLE `orc_two_stripes_dict_and_nodict`(
  `x` int,
  `y` string)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
  'hdfs://emr-header-1.cluster-49091:9000/user/hive/warehouse/zya.db/orc_two_stripes_dict_and_nodict'
TBLPROPERTIES (
  'transient_lastDdlTime'='1682014085')

And here is the wrong result

I've verified that your pr works. I've approved that.
But if you can add some UT, it would be better.
Can you take a look and see if you can add a unit test with this code? The unit test should load the 'hdfs_scanner_test.cpp' file, which contains several test cases for ORC (Optimized Row Columnar). I can send you the orc file I generated.

@kobebryantlin0 Thank you for your contribution, could you clang-format your code and sign-off by git commit -s

@kobebryantlin0 Thank you for your contribution, could you clang-format your code and sign-off by git commit -s

Sure, I can do that, but it may take a while as I'm currently at work.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: kobebryantlin0:x: linenwei1linenwei1 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

@kobebryantlin0 Thanks for the fix. Could you add a unit test.

@DorianZheng @Smith-Cruise ut have been added.

All checks have passed

@DorianZheng @imay all checks have passed, could you help me merge pr into master? thx.

@Mergify backport branch-3.0


Interrupt currently playing animation to immediately play a new one?

I have noticed that if I try to play an animation with a host that is currently playing an animation, nothing happens.  I can't play any new animations until the existing one stops.  Is there a way to stop the existing animation so I can immediately play a new one?
Also, a giant thank you for creating this library.  I have used it to create an automated, animated improv theater that uses OpenAI's  GPT-3 natural language generation system to create animated scenes in real-time based on scene suggestions.  If you want to see an example of this, check out this video:
https://www.youtube.com/watch?v=QL4QQ5NsPL8
Again, thank you for such a well designed and innovative software library!

Hey @roschler, thanks for showing us what you have used this library to make!
With respect to your question, are you referring to the GestureFeature.playGesture() animations? If so there is a force option that can be set in the call options. This will cause the animation to play regardless of any currently playing animations:
https://github.com/aws-samples/amazon-sumerian-hosts/blob/mainline/src/core/GestureFeature.js#L537


photoPickerTitle is not working

After checking the code, different name 'pickerTitle' is used for the props 'photoPickerTitle':
options = {
    title: this.props.pickerTitle || 'Select Photo',
    storageOptions: {
      skipBackup: true,
      path: 'images'
    }
  }

Please fix, thanks!

Fixed, Thank you!


Loading parametrs from others configuration file

We would like to have the ability to launch application with custom parameters like -Xmx, etc. In order to do it, we need to consider the ability to launch the application with another configuration file (not dbeaver.ini)

Connected to
#7727
#7546
#21806
#16187
#12683
#35080

This will be implemented here: https://github.com/dbeaver/dbeaver/issues/35080


Add heatwave3

Insert URLs to the project here: https://github.com/robwschlegel/heatwave3

[x] The projects is active, documented, open source licensed, shows usage from external parties and is directly targeting environmental sustainability. Find more details in the Contribution Guide.

All issues labeled as Good First Issue of the project listed on OpenSustain.tech will be visible on ClimateTriage.com. This is a great way to welcome new community members to your project.
All projects listed will be posted on our Mastodon channel

@robwschlegel


Piglin Bartering Pack Books

We're running the Rebalanced Piglin Bartering [PurpurPack] V3.0 & recently updated V3.1 (both having the same issue).
Problem:  Running the Books in the Loot table gives ALL enchanted books.
"type": "minecraft:item",
"name": "minecraft:book",
"weight": 5,
"functions": [
{
"function": "minecraft:enchant_randomly",
"enchantments": [
"minecraft:soul_speed"
Expected behavior: Bartering should only give Soul Speed Books 1-3.
Solution for now: Deleted Books from the loot table.

Fix has been pushed, and should be available now:
https://modrinth.com/datapack/purpurpacks-rebalanced-piglin-bartering/version/3.2
Thanks for reporting!