Chat History example

Hi, can you please provide some examples to use chat history with chatcompletition?
Thanks

Just a suggestion :D

I would do it this way using the prompt object itself!

	private void openAiChatRunUseInstruction() {
		List<Message> messages = new ArrayList<>();
		Prompt prompt = new Prompt(messages);

		sendPromptAndUpdate(prompt, "What is the population of Canada?");
		sendPromptAndUpdate(prompt, "What is the capital?");

		prompt.getInstructions().forEach(System.out::println);

	}

	private void sendPromptAndUpdate(Prompt prompt, String userPrompt) {
		prompt.getInstructions().add(new UserMessage(userPrompt));
		ChatResponse resp = openAiChatClient.call(prompt);
		prompt.getInstructions().add(resp.getResult().getOutput());
	}

Output:

UserMessage{content='What is the population of Canada?', properties={}, messageType=USER}
AssistantMessage{content='As of 2021, the population of Canada is approximately 38 million.', properties={role=ASSISTANT}, messageType=ASSISTANT}
UserMessage{content='What is the capital?', properties={}, messageType=USER}
AssistantMessage{content='The capital of Canada is Ottawa.', properties={role=ASSISTANT}, messageType=ASSISTANT}

Or maybe this way using new container to organize output ?

	private void openAiChatRunWithList() {
		List<Message> messages = new ArrayList<>();

		String userPrompt1 = "What is the population of Canada?";
		addMessageAndCallApi(messages, userPrompt1);

		String userPrompt2 = "And what is the capital (only name)?";
		addMessageAndCallApi(messages, userPrompt2);

		messages.forEach(System.out::println);
		/**
		 * UserMessage{content='What is the population of Canada?', properties={}, messageType=USER}
		 * AssistantMessage{content='As of 2021, the estimated population of Canada is around 38 million.', properties={role=ASSISTANT}, messageType=ASSISTANT}
		 * UserMessage{content='And what is the capital (only name)?', properties={}, messageType=USER}
		 * AssistantMessage{content='The capital of Canada is Ottawa.', properties={role=ASSISTANT}, messageType=ASSISTANT}
		 */
	}

	private void addMessageAndCallApi(List<Message> messages, String userPrompt) {
		messages.add(new UserMessage(userPrompt));
		Prompt prompt = new Prompt(messages);
		ChatResponse resp = openAiChatClient.call(prompt);
		resp.getResults().forEach(r -> {
			messages.add(r.getOutput());
		});
	}

Output:

UserMessage{content='What is the population of Canada?', properties={}, messageType=USER}
AssistantMessage{content='As of 2021, the estimated population of Canada is around 38 million.', properties={role=ASSISTANT}, messageType=ASSISTANT}
UserMessage{content='And what is the capital (only name)?', properties={}, messageType=USER}
AssistantMessage{content='The capital of Canada is Ottawa.', properties={role=ASSISTANT}, messageType=ASSISTANT}

Hi @hemeda3 , we will start working on an official Spring AI ChatHistory abstraction very soon.
Until then perhaps you can use something like this: https://github.com/tzolov/playground-flight-booking/blob/main/src/main/java/ai/spring/demo/ai/playground/services/ChatHistory.java

I don't think there is any implicit support for chat for now, the idea is to send the previous discussion as part of the prompt itself and this helps the chat bot maintain the context of the application. I made a video in details on this topic where I built a chat bot using open ai chat completion API. Here is the video if you wish to watch https://www.youtube.com/watch?v=83KTwxCCe9I
Moreover, please note that LLMs are stateless by default and by design. So all the chatbots like chat gpt or Gemini they implement the context awareness in the message thread on the application level. So effectively, maintaining the chat context/awareness/history over a LLM is a matter of having a wrapper of some sort. Chat GPT web portal does so by sending something called as a conversation_id

I don't think there is any implicit support for chat for now, the idea is to send the previous discussion as part of the prompt itself and this helps the chat bot maintain the context of the application. I made a video in details on this topic where I built a chat bot using open ai chat completion API. Here is the video if you wish to watch https://www.youtube.com/watch?v=83KTwxCCe9I
Moreover, please note that LLMs are stateless by default and by design. So all the chatbots like chat gpt or Gemini they implement the context awareness in the message thread on the application level. So effectively, maintaining the chat context/awareness/history over a LLM is a matter of having a wrapper of some sort. Chat GPT web portal does so by sending something called as a conversation_id

One downside of this approach is accumulation of large amount of tokens quickly which is both bad of being within  the context window limit and the billing. However, one common optimisation is to either summarise the discussion so far (by using the chat api itself) so the context text becomes compact and smaller yet meaningful enough to convey the context or maintain context for only last K messages in a given session (chat window which is tracked by web ui of chat gpt). I have discussed all these information in more details in the video linked above.

Isn't the purpose of feature like "thread" provided by OpenAI? I guess abstraction could use it as implementation (or in-memory list of history messages, should it be decided by configuration?).

I don't think there is any implicit support for chat for now, the idea is to send the previous discussion as part of the prompt itself and this helps the chat bot maintain the context of the application. I made a video in details on this topic where I built a chat bot using open ai chat completion API. Here is the video if you wish to watch https://www.youtube.com/watch?v=83KTwxCCe9I
Moreover, please note that LLMs are stateless by default and by design. So all the chatbots like chat gpt or Gemini they implement the context awareness in the message thread on the application level. So effectively, maintaining the chat context/awareness/history over a LLM is a matter of having a wrapper of some sort. Chat GPT web portal does so by sending something called as a conversation_id

One downside of this approach is accumulation of large amount of tokens quickly which is both bad of being within the context window limit and the billing. However, one common optimisation is to either summarise the discussion so far (by using the chat api itself) so the context text becomes compact and smaller yet meaningful enough to convey the context or maintain context for only last K messages in a given session (chat window which is tracked by web ui of chat gpt). I have discussed all these information in more details in the video linked above.

need Assistants API

Spring AI 1.0.0-M1 introduced support for chat history via the new ChatClient API: https://docs.spring.io/spring-ai/reference/api/chatclient.html#_chat_memory You can find an example here: https://github.com/ThomasVitale/llm-apps-java-spring-ai/tree/main/00-use-cases/chatbot
@williamspindox Can this issue be closed?

Closing, please reopen if necessary @williamspindox


Signature v4 on a multipart upload

Hello.
I am glad to see this plugin. Thanks again.
This is more my ignorance question.
I am trying to follow this https://github.com/arnemolland/sigv4/blob/master/example/example.dart
I am trying to post an video file.
Here is what I get from s3 pre-signed url request:
{fields: {AWSAccessKeyId: <>, key: <>, policy:<POLICY>, signature: <>}}
I am constructing my request object like this:
      http.MultipartFile multipartFile = await http.MultipartFile.fromPath('image_file', filepath);
      request.files.add(multipartFile);
      request.fields['key'] = signed_url['fields']['key'];
      request.fields['AWSAccessKeyId'] = signed_url['fields']['AWSAccessKeyId'];
      request.fields['policy'] = signed_url['fields']['policy'];
      request.fields['signature'] = signed_url['fields']['signature'];
      request.fields['file'] = filepath;

from the example:
final client = Sigv4Client(
    accessKey: 'your_access_key',
    secretKey: 'your_secret_key',
  );

I am following this doc to generate the url - https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html
I have couple of questions:

I can see how to get the access key but I dont see a secret key. Is it part of the policy? what field represents the secretKey?
the client as defined on the example, comes with a error The named parameter 'accessKey' isn't defined.

Appreciate a response. I have been working on this issue for the past couple of weeks.
Shekar

I tried using the constructor this way:
final client = Sigv4Client(
        signed_url['fields']['AWSAccessKeyId'],
        signed_url['fields']['key'],
        signed_url['url']
      );

but when I try ```
// A larger request
final request = client.request(
'https://service.aws.com/endpoint',
method: 'POST',
queryParameters: {'key': 'value'},
headers: {'header': 'value'},
body: {'content': 'some-content'},
);
I get an error saying ``` The setter 'request' isn't defined for the class 'Sigv4Client'. ```

Hi @ctippur 👋
If anything below is unclear, check out the AWS documentation.
The accessKey parameter refers to the Access Key ID, and the secretKey parameter refers to the secret access key itself. I'll update the API to name these more clearly.
Both the accessKey and secretKey parameters are required. The example provided does not run without providing your own accessKey, secretKey and endpoint. The existing values are placeholders, and the service.aws.com/endpoint path is purely fictional. I'll update the docs to make this clearer.
I'm unsure about the last bit though. It looks like you're trying to set a request field in the client, but there's only a request method. Do you have a complete example of this part?

@arnemolland thanks for responding. Sorry about the delay.
My bad on the request. I was pulling the wrong version.
Now that the request method is sorted, can you please give a little more clarity on the large request object
final largeRequest = client.request(
    path,
    method: 'POST',
    query: {'key': 'value'},
    headers: {'header': 'value'},
    body: {'content': 'some-content'},
  );

  // POST request
  post(largeRequest.url, headers: largeRequest.headers, body: largeRequest.body);

path here refers to the path to the local file?
query - can you please give a sample for what should go here
header - I am guessing we can add the Content-Type here
body - what should go here?

Thanks again.
Shekar

Just to be a little more explicit -
The objective is to upload an image or a video file using flutter.
I have presigned URL object via generate_presigned_post call. The object i have looks like this:
{fields: {AWSAccessKeyId: <>, key: <>, policy:<POLICY>, signature: <>}}
I am struggling to unravel the right mapping to do a POST call.
I have created a client
final  client = Sigv4Client(
      accessKey: accessKeyId,
      secretKey: _secretKeyId,
      region: _region,
      );

// Create a local file
var lf1 = await http.MultipartFile.fromPath('image_file', filepath);
var largeRequest;
try{
 largeRequest = client.request(
      _s3Endpoint,
      method: 'POST',
      headers: {'Content-Type': 'Application/octet-stream'},
      body: {'file': lf1}
    );
} catch (e) {
  print ("Large request error: " + e.toString());
}

  // POST request
  try{
    var response = await http.post(largeRequest.url, headers: largeRequest.headers, body: largeRequest.body);
    print (response.body);
  }catch (e){
    print (e.toString());
  }

I get an error
flutter: Large request error: Converting object to an encodable object failed: Instance of 'MultipartFile'

Thanks for the details @ctippur
To answer your questions:

path refers to the absolute path of your endpoint, e.g. https://service.com/api/list
query is a map of query parameters, and is optional
headers is the request headers and is optional as well. The Content-Type header defaults to either what you set as default when instantiating your Sigv4Client with the parameter defaultContentType, which again defaults to application/json if omitted
body (or payload) is the request message body, and is optional. This is often used with POST or PATCH requests.

For your error, you're providing some object to the body field. This object has to be encodable, which means that its fields need to be directly encodable to a JSON string. These types are encodable:

number
boolean
string
null
list
map with string keys

If your object is non-encodable by default, you can encode it "manually" using dart:convert:
// This is just an example and will not work in your case
final encoded = jsonEncode(value, toEncodable: (value) => value.toString());

If toEncodable is omitted, the JSON encoder tries calling a toJson() method on the object. If none exists, it will throw an error, which is what you're getting.
MultiPartFile is not encodable and does not expose a toJson() method. To solve this, you could pass the data as a byte array (Uint8List is encodable, as it is a list):
// MultiPartFile => Uint8List
final bytes = await file.finalize().toBytes()
...
largeRequest = client.request(
  _s3Endpoint,
  method: 'POST',
  headers: {'Content-Type': 'Application/octet-stream'},
  body: bytes,
);

@arnemolland
Thanks for the explanation
I followed your instructions.
This is what I have:
final bytes = await lf1.finalize().toBytes();
final  client = Sigv4Client(
      accessKey: accessKeyId,
      secretKey: _secretKeyId,
      region: _region,
      );

var largeRequest;
    try {
      largeRequest = client.request(
        _s3Endpoint,
        method: 'POST',
        headers: {'Content-Type': 'Application/octet-stream'},
        body: bytes,
      );
    }catch (e, s) {
      print ("Large request error " + e.toString());
      print ("Large request stack " + s.toString());
    }

Here is the error:
flutter: Large request error Invalid argument(s)
flutter: Large request stack #0      _StringBase.+  (dart:core-patch/string_patch.dart:263:57)
#1      Sigv4.buildAuthorizationHeader
package:sigv4/src/sigv4.dart:133
#2      Sigv4Client._generateAuthorization
package:sigv4/src/client.dart:198
#3      Sigv4Client.signedHeaders
package:sigv4/src/client.dart:116
#4      Sigv4Client.request
package:sigv4/src/client.dart:149
#5      TestsBloc.update_signed_url
package:helloworld/…/testsBloc/tests_bloc.dart:361
<asynchronous suspension>

Something is fishy. I am now getting a different error:

For the first issue, it seems to me like there is an invalid Access Key ID, Secret Access Key or region passed to the client. I’ll improve the error handling for this case.
For the OSException, it seems like you’re making a request to my s3 bucket which is an invalid URL and will not reach a server.

my s3 bucket is a url of the form https://<my bucket>.s3.amazonaws.com/

Here is the tcpdump from the transaction
23:13:58.283118 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R.], seq 4177, ack 21788, win 287, length 0
23:13:58.284111 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq<PHONE_NUMBER>, win 0, length 0
23:13:58.284117 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq<PHONE_NUMBER>, win 0, length 0
23:13:58.284118 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq<PHONE_NUMBER>, win 0, length 0
23:13:58.284120 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq<PHONE_NUMBER>, win 0, length 0
23:13:58.284121 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq<PHONE_NUMBER>, win 0, length 0
23:13:58.284122 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq<PHONE_NUMBER>, win 0, length 0
23:13:58.284123 IP s3-w.ap-south-1.amazonaws.com.https > shekars-mbp.62441: Flags [R], seq<PHONE_NUMBER>, win 0, length 0
As we can see the server is sending a reset back :/

Meanwhile, I am trying a put variant of the presigned url
Put variant gives a URL
 https://My_bucket.s3.amazonaws.com/?AWSAccessKeyId=<ACCESS_KEY>&Signature=<Signature>&x-amz-security-token=<SEC_TOKEN>&Expires=<Expires>

Here is my code for that:
// Create a client
final client = Sigv4Client(
      secretKey: uri.queryParameters['Signature'],
      accessKey: uri.queryParameters['AWSAccessKeyId'],
      defaultContentType: 'application/octet-stream'
    );

   // Create a request object
    final request = client.request(
      'https://' + uri.host + '/' + uri.path,
      method: 'PUT',
      //query: {},
      headers: {'Content-Type': 'application/octet-stream', 'x-amz-content-sha256': "UNSIGNED-PAYLOAD"},
      body: {'content': 'some-content'},
    );

   // Make a http put request
   try {

      var response = await http.put(request.url, headers: request.headers, body: request.body);
      print (response.body);

    }catch (e,s){

      print ("**** Error **** " + e.toString());
      print ("**** Stack **** " + s.toString());

    }

Error I am getting is
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; incorrect service "execute-api". This endpoint belongs to "s3".</Message><RequestId>010975D5EE84ED8B</RequestId><HostId>Host id redacted</HostId></Error>

I am guessing I should be encouraged by this error.

I should’ve spotted this earlier. The service parameter of your client should be set to s3. It defaults to execute-api, which is why S3 is closing the connection.

Aha. Thx
Changed that. Now a different error:
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>REDACTED</AWSAccessKeyId><RequestId>EA78B8A76190CAA9</RequestId><HostId>REDACTED</HostId></Error>

Meanwhile, Let me switch the call to POST and make the corresponding service change as well and try.

Just tested POST after making the changes to client
final  client = Sigv4Client(
      accessKey: _accessKeyId,
      secretKey: _secretKeyId,
      region: _region,
      defaultContentType: "application/octet-stream",
      serviceName: 's3'
      );

I am still getting the same error:
SocketException: OS Error: Connection reset by peer, errno = 54, address = <MY bucket>, port = 50301

I should’ve spotted this earlier. The service parameter of your client should be set to s3. It defaults to execute-api, which is why S3 is closing the connection.
I’ll update the constructor to make it a required parameter without a default value.

I agree..

Could this  be a content type issue?
for a post request, the content type should be multipart/form-data.
I tried changing the POST call code to
final  client = Sigv4Client(
      accessKey: _accessKeyId,
      secretKey: _secretKeyId,
      region: _region,
      defaultContentType: 'multipart/form-data',
      serviceName: 's3'
      );

    var largeRequest;
    try {
      largeRequest = client.request(
        _s3Endpoint,
        method: 'POST',
        body: bytes,
      );
    }catch (e, s) {
      print ("Large request error " + e.toString());
      print ("Large request stack " + s.toString());
    }

Got the same error

Since you're making a multipart request, you should try using MultipartRequest, as the request() method only returns a regular Request object.

To keep this simple, would something like this work?
var config = File('config.txt');
var bytes = await config.readAsBytes();

And use the body as bytes?

Since you're making a multipart request, you should try using MultipartRequest, as the request() method only returns a regular Request object.

Bit confused with this.
Looking at  MultipartRequest, the request object is a http.MultipartRequest type.
I dont need it to be MultipartRequest, I can make do with a simpler solution.

Yes, any encodable primitive or object can be used in the body. It’s up to you what request classes you’d like to use, the only thing you need from this library is the signed request headers you can get using the getSignedHeaders() method. The request() method is just a wrapper that puts them in a Request object.

Yes, any encodable primitive or object can be used in the body. It’s up to you what request classes you’d like to use, the only thing you need from this library is the signed request headers you can get using the getSignedHeaders() method. The request() method is just a wrapper that puts them in a Request object along with a canonical URI.

Possible to give an example? client object doesnt seem to have getSignedHeaders() method.

I had a typo, sorry, it's signedHeaders(). The error message you're getting is pretty descriptive, as you're probably passing some dynamic value where it's expecting a <String, String> map. An example of how it might look:
  final file = MultipartFile.fromString('field', 'value');
  final multipart = MultipartRequest('PUT', Uri.parse(path))..files.add(file);
  final bytes = await file.finalize().toBytes();

  final signed = client.signedHeaders(path, method: 'PUT', body: bytes);

  multipart.headers.addAll(signed);

  final request = await multipart.send();

I had a typo, sorry, it's signedHeaders(). The error message you're getting is pretty descriptive, as you're probably passing some dynamic value where it's expecting a <String, String> map. An example of how it might look:
  final file = MultipartFile.fromString('field', 'value');
  final multipart = MultipartRequest('PUT', Uri.parse(path))..files.add(file);
  final bytes = await file.finalize().toBytes();

  final signed = client.signedHeaders(path, method: 'PUT', body: bytes);

  multipart.headers.addAll(signed);

  final request = await multipart.send();

Getting into some syntax errors at final multipart = MultipartRequest('PUT', Uri.parse(path))..files.add(file);
MultipartFile file
The argument type 'MultipartFile (where MultipartFile is defined in /Users/shekar/playground/flutter/flutter/.pub-cache/hosted/pub.dartlang.org/dio-3.0.7/lib/src/multipart_file.dart)' can't be assigned to the parameter type 'MultipartFile (where MultipartFile is defined in /Users/shekar/playground/flutter/flutter/.pub-cache/hosted/pub.dartlang.org/http-0.12.0+2/lib/src/multipart_file.dart)'.dart

You’re importing both dio and http which both expose a MultiPartFile. Add a prefix to resolve any ambiguity.
import ‘package:http/http.dart’ as http;

var file = http.MultiPartFile(...)

I had a typo, sorry, it's signedHeaders(). The error message you're getting is pretty descriptive, as you're probably passing some dynamic value where it's expecting a <String, String> map. An example of how it might look:
  final file = MultipartFile.fromString('field', 'value');
  final multipart = MultipartRequest('PUT', Uri.parse(path))..files.add(file);
  final bytes = await file.finalize().toBytes();

  final signed = client.signedHeaders(path, method: 'PUT', body: bytes);

  multipart.headers.addAll(signed);

  final request = await multipart.send();

My apologies about the syntax error. I tried this and I still get the same error:
type '_InternalLinkedHashMap<String, dynamic>' is not a subtype of type 'Map<String, String>'

Are you om the latest version of sigv4? signedHeaders() should return a <String, String> map.
HTTP headers always contain string values only, which is why the Request object does not allow any dynamic values. If you want to include a non-encodable object, you need to convert is to a JSON-string, e.g. using jsonEncode from the dart:convert library.
The AWS Signature that this package helps generate is purely headers that contain information about the request itself. The signature is used to check the integrity of the request data. This means that you need to include your other request data like body, headers and query parameters when generating these signed headers.
You can then add these headers to a request, however the request itself must match the request described when generating the signed headers. If you either add or remove some data, you’ll break the integrity of the request, the signature will not match and AWS will reject it.
This is why you will need to include your bytes value both in the request itself and when generating the signed headers. If not, the signature will not match the request.
Here’s a link to the AWS Signature v4 documentation if anything of this was unclear.

I tried this. Tried to avoid body: bytes from signed
    var url_s3 = 'https://' + uri.host  + uri.path + '/';
    var uri_s3 = Uri.parse('https://' + uri.host + '/' + uri.path,);

    final file = await http.MultipartFile.fromString('videofile',filepath);
    final multipart = http.MultipartRequest('PUT', uri_s3)..files.add(file);
    var bytes;
    try{
      bytes = await file.finalize().toBytes();
    }catch (e){
      print ("**** Exception in getting bytes **** " + e.toString());
    }

    var signed;
    try {
      signed = client.signedHeaders(url_s3, method: 'PUT');
    } catch (e) {
      print ("*** Exception in signing *** " + e.toString());
    }

    print ("**** Add headers to to multipart ****" );
    Map<String,String> signed_map;

    try {
      multipart.headers.addAll(signed);
    }catch (e) {
      print ("*** Exception in adding headers *** " + e.toString());
    }

    print ("**** Send request ****" );

    try{
      final request = await multipart.send();
      //print (request.statusCode);
    }catch (e){
      print ("**** EXCEPTION ON PUSING MULTIPARTREQUEST " + e.toString());
    }

The exception I get is Unhandled Exception: Bad state: Can't finalize a finalized MultipartFile.

Are you om the latest version of sigv4? signedHeaders() should return a <String, String> map.
HTTP headers always contain string values only, which is why the Request object does not allow any dynamic values. If you want to include a non-encodable object, you need to convert is to a JSON-string, e.g. using jsonEncode from the dart:convert library.
The AWS Signature that this package helps generate is purely headers that contain information about the request itself. The signature is used to check the integrity of the request data. This means that you need to include your other request data like body, headers and query parameters when generating these signed headers.
You can then add these headers to a request, however the request itself must match the request described when generating the signed headers. If you either add or remove some data, you’ll break the integrity of the request, the signature will not match and AWS will reject it.
This is why you will need to include your bytes value both in the request itself and when generating the signed headers. If not, the signature will not match the request.
Here’s a link to the AWS Signature v4 documentation if anything of this was unclear.

Thanks for the detailed explanation. I do have the latest version.
With all the changes, I think the last straw in the puzzle maybe to resolve the exception
 Unhandled Exception: Bad state: Can't finalize a finalized MultipartFile.

You could check it before trying to finalize:
var bytes;

If(!file.finalized) {
  bytes = await file.finalize().toBytes();
} else {
  bytes = file.toBytes();
}

I’m on my phone, so take this as pseudocode.

Thanks @arnemolland. bytes = file.toBytes(); wont work as it is a MultipartFile

tried to simplify it by taking away multipartfile.
   final client = Sigv4Client(
      keyId: uri.queryParameters['Signature'],
      accessKey: uri.queryParameters['AWSAccessKeyId'],
      defaultContentType: 'application/octet-stream',
      region: 'ap-south-1',
      serviceName: 's3',
    );
  final bytes = File(filepath).readAsBytesSync();
  final  signed = client.signedHeaders(signed_url, method: 'PUT', body: bytes);
  signed.addAll({'x-amz-content-sha256': "UNSIGNED-PAYLOAD"});
  final simpleput = await http.put(
      url_s3,
      headers: signed,
      body: bytes
      );

    print (simpleput.body);

The error I get is
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the authorization component "Credential=SIGNATURE/20191217/ap-south-1/s3/aws4_request" is malformed.</Message><RequestId>B7F60499DDF13FF3</RequestId><HostId>Host id</HostId></Error>

I am not sure if I am using keyId correctly.
keyId: uri.queryParameters['Signature'],
Referring back to the signed url structure - ``` https://My_bucket.s3.amazonaws.com/?AWSAccessKeyId=<ACCESS_KEY>&Signature=&x-amz-security-token=<SEC_TOKEN>&Expires=

Just for completion, here is the code I have to generate signed url
s3_client = boto3.client('s3')
        try:
            response = s3_client.generate_presigned_url('put_object',
                                                        Params={'Bucket': bucket_name,
                                                                'Key': object_name},
                                                        ExpiresIn=expiration)
        except ClientError as e:
            logging.error("In client error exception code")
            logging.error(e)
            return None

        # The response contains the presigned URL
        return response

@arnemolland I got more clarity on this. For .a put call, the code from the server side should generate a url that has all the components in it. As we are using presigned url, the ensuing url can be used as is without any modifications.
SignatureV4 is needed for aws api access from other sources. I am sorry to have taken you through this path but it was a learning curve for me. I was able to resolve this issue after adding some additional parameters to the server code that renders the url. Please refer to the stackoverflow link for details.

Great! The important thing is that you’ve solved it and gained experience.


Plans & Feature Availability

Per this announcement: https://www.microsoft.com/en-us/microsoft-365/blog/2019/01/02/myanalytics-the-fitness-tracker-for-work-is-now-more-broadly-available/
There's not a specific explanation or delineation in the announcement between which features (as shown in the documentation above) will be available to what plans. Is the table here accurate in light of the upcoming changes?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8f1b78b4-21ad-2234-0822-149ffe88814b
Version Independent ID: 5dad7085-6921-5392-f0a7-f5b6e9b5cc0b
Content: MyAnalytics plans and environments - Workplace Analytics Office 365
Content Source: WorkplaceAnalytics/MyAnalytics/Overview/plans-environments.md
Product: mya
GitHub Login: @paul9955
Microsoft Alias: v-pascha

Yes, the plans and features described here are accurate. Please refer to the product blog for any updated information about upcoming changes.


Outline shows duplicated "Jupyter" and "Python" entries

The outline view in VSCode has recently started doing something mildly annoying where the outline entries are listed twice when I am editing a .py file, if I have a Jupyter notebook open at the same time.
I submitted this as a bug to the VSCode repo but they told me to bring it here.
Environment data

VS Code version: 1.69.2
Jupyter Extension version (available under the Extensions sidebar): v2022.6.1201981810
Python Extension version (available under the Extensions sidebar): v2022.10.1
OS (Windows | Mac | Linux distro) and version: Windows_NT x64 10.0.19043
Python and/or Anaconda version: Python 3.9.7
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
Jupyter server running: Local

Expected behaviour
The outline view should only display one set of entries.
Actual behaviour
Two entries are displayed, see screenshot.

Steps to reproduce:
[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Open a .py file. Outline in the Explorer looks normal (displays classes, functions, etc).

Open a .ipynb notebook file.  Outline displays Jupyter markdown cells, looks normal.

Switch back to the .py file. Outline now has "Jupyter" and "Python" headers for this .py file, which contains duplicated entries (same entries as in step 1: classes, functions, etc).

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

Visual Studio Code (1.69.2, undefined, desktop)
Jupyter Extension Version: 2022.6.1201981810.
Python Extension Version: 2022.10.1.
No workspace folder opened.
info 12:46:4.907: ZMQ install verified.
User belongs to experiment group 'jupyterTest'
User belongs to experiment group 'jupyterEnhancedDataViewer'
info 12:46:5.350: LSP Notebooks experiment is disabled -- not in treatment group
info 12:46:5.575: Experiment status for python is {"enabled":true,"optInto":[],"optOutFrom":[]}
info 12:46:6.155: Process Execution: > c:\Program Files (x86)\Python310-32\python.exe -m pip list

c:\Program Files (x86)\Python310-32\python.exe -m pip list
info 12:46:18.257: Preferred Remote kernel for f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb is undefined
info 12:46:18.275: Preferred Remote kernel for f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb is undefined
info 12:46:18.275: Find preferred kernel for f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb with metadata {"kernelspec":{"display_name":"Python 3.9.7 ('acid')","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.7"},"orig_nbformat":4,"vscode":{"interpreter":{"hash":"986d8e3c41ce8164780d90ff3d0b15bc8797f31c4543868abad1dfa1775e5138"}}} & preferred interpreter c:\Program Files (x86)\Python310-32\python.exe
info 12:46:18.278: Preferred Remote kernel for f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb is undefined
info 12:46:18.278: Preferred kernel .jvsc74a57bd0986d8e3c41ce8164780d90ff3d0b15bc8797f31c4543868abad1dfa1775e5138.c:\ProgramData\miniforge3\envs\acid\python.exe.c:\ProgramData\miniforge3\envs\acid\python.exe.-m#ipykernel_launcher is exact match
info 12:46:18.278: PreferredConnection: .jvsc74a57bd0986d8e3c41ce8164780d90ff3d0b15bc8797f31c4543868abad1dfa1775e5138.c:\ProgramData\miniforge3\envs\acid\python.exe.c:\ProgramData\miniforge3\envs\acid\python.exe.-m#ipykernel_launcher found for NotebookDocument: f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb
info 12:46:18.314: Preferred Remote kernel for f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb is undefined
info 12:46:18.314: Find preferred kernel for f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb with metadata {"kernelspec":{"display_name":"Python 3.9.7 ('acid')","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.7"},"orig_nbformat":4,"vscode":{"interpreter":{"hash":"986d8e3c41ce8164780d90ff3d0b15bc8797f31c4543868abad1dfa1775e5138"}}} & preferred interpreter c:\Program Files (x86)\Python310-32\python.exe
info 12:46:18.317: Preferred Remote kernel for f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb is undefined
info 12:46:18.317: Preferred kernel .jvsc74a57bd0986d8e3c41ce8164780d90ff3d0b15bc8797f31c4543868abad1dfa1775e5138.c:\ProgramData\miniforge3\envs\acid\python.exe.c:\ProgramData\miniforge3\envs\acid\python.exe.-m#ipykernel_launcher is exact match
info 12:46:18.776: Registering dummy command feature
info 12:46:18.818: Starting Jupyter Session id = 'startUsingPythonInterpreter:.jvsc74a57bd0986d8e3c41ce8164780d90ff3d0b15bc8797f31c4543868abad1dfa1775e5138.c:\ProgramData\miniforge3\envs\acid\python.exe.c:\ProgramData\miniforge3\envs\acid\python.exe.-m#ipykernel_launcher' (Python Path: c:\ProgramData\miniforge3\envs\acid, EnvType: Conda, EnvName: 'acid', Version: 3.9.7) for 'f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb' (disableUI=true)
info 12:46:18.818: Creating raw notebook for resource 'f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb'
info 12:46:18.818: Computing working directory for resource 'f:\temp\Joel K temp\Python stuff\dummy notebook.ipynb'
info 12:46:18.857: Starting raw kernel 'acid (Python 3.9.7)' for interpreter c:\ProgramData\miniforge3\envs\acid\python.exe
info 12:46:18.913: Kernel launching with ports 9000,9001,9002,9003,9004. Start port is 9000
info 12:46:19.809: Registering dummy command feature
info 12:46:23.286: Process Execution: > c:\ProgramData\miniforge3\envs\acid\python.exe -m pip list
c:\ProgramData\miniforge3\envs\acid\python.exe -m pip list
info 12:46:23.289: Process Execution: > c:\ProgramData\miniforge3\envs\acid\python.exe -c "import ipykernel; print(ipykernel.version); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.file)"
c:\ProgramData\miniforge3\envs\acid\python.exe -c "import ipykernel; print(ipykernel.version); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.file)"
info 12:46:23.295: Creating daemon process for c:\ProgramData\miniforge3\envs\acid\python.exe with env variables count 64
info 12:46:23.298: Process Execution: > c:\ProgramData\miniforge3\envs\acid\python.exe -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_interrupt_daemon -v --ppid 11096
c:\ProgramData\miniforge3\envs\acid\python.exe -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_interrupt_daemon -v --ppid 11096
info 12:46:23.562: get interrupthandle daemon
info 12:46:23.566: Process Execution: > c:\ProgramData\miniforge3\envs\acid\python.exe -m ipykernel_launcher --ip=<IP_ADDRESS> --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"b9b7b7a6-4cbc-4d5b-b66a-8a6f994ccb61" --shell=9002 --transport="tcp" --iopub=9004 --f=c:\Users\joelk\AppData\Roaming\jupyter\runtime\kernel-v2-11096cjuNidOe9sfd.json
c:\ProgramData\miniforge3\envs\acid\python.exe -m ipykernel_launcher --ip=<IP_ADDRESS> --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"b9b7b7a6-4cbc-4d5b-b66a-8a6f994ccb61" --shell=9002 --transport="tcp" --iopub=9004 --f=c:\Users\joelk\AppData\Roaming\jupyter\runtime\kernel-v2-11096cjuNidOe9sfd.json
info 12:46:23.566: Process Execution: cwd: f:\temp\Joel K temp\Python stuff
cwd: f:\temp\Joel K temp\Python stuff
info 12:46:23.810: ipykernel version 6.5.0 for c:\ProgramData\miniforge3\envs\acid\python.exe
info 12:46:23.810: ipykernel location c:\ProgramData\miniforge3\envs\acid\lib\site-packages\ipykernel_init_.py for c:\ProgramData\miniforge3\envs\acid\python.exe
warn 12:46:24.979: StdErr from Kernel Process c:\ProgramData\miniforge3\envs\acid\lib\site-packages\traitlets\traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
warn(
c:\ProgramData\miniforge3\envs\acid\lib\site-packages\traitlets\traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'b9b7b7a6-4cbc-4d5b-b66a-8a6f994ccb61' instead of 'b"b9b7b7a6-4cbc-4d5b-b66a-8a6f994ccb61"'.
warn(

info 12:46:25.56: Kernel Output: NOTE: When using the ipython kernel entry point, Ctrl-C will not work.
To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.
To read more about this, see https://github.com/ipython/ipython/issues/2049
To connect another client to this kernel, use:
--existing c:\Users\joelk\AppData\Roaming\jupyter\runtime\kernel-v2-11096cjuNidOe9sfd.json
info 12:46:25.99: Started kernel acid (Python 3.9.7)., (Raw session started and connected)
info 12:46:25.100: Finished connecting 4073ffc0-c1df-4c0b-984b-49ab60b02cab
info 12:46:25.102: UpdateWorkingDirectoryAndPath in Kernel
info 12:46:25.102: Executing silently Code (idle) = import os\nimport sys\n%cd "f:\temp\Joel K temp\Python stuff"\nif os.getcwd() not in sys.path:\nsys.
info 12:46:25.131: Executing silently Code (completed) = import os\nimport sys\n%cd "f:\temp\Joel K temp\Python stuff"\nif os.getcwd() not in sys.path:\nsys. with 1 output(s)
info 12:46:25.131: Waiting for idle on (kernel): 18a2e493-1728-4626-b494-fe0fb5f8daad -> idle
info 12:46:25.131: Finished waiting for idle on (kernel): 18a2e493-1728-4626-b494-fe0fb5f8daad -> idle
info 12:46:25.143: Widget Message: Received IPyWidgets_kernelOptions in ScriptManager
info 12:46:25.143: Widget Message: Received IPyWidgetMessages.IPyWidgets_kernelOptions
error 12:46:25.171: Failed to get baseUrl [Error: ENOENT: no such file or directory, lstat 'c:\ProgramData\miniforge3\envs\acid\share\jupyter\nbextensions'] {
errno: -4058,
code: 'ENOENT',
syscall: 'lstat',
path: 'c:\ProgramData\miniforge3\envs\acid\share\jupyter\nbextensions'
}

dup of https://github.com/microsoft/pylance-release/issues/3027

I see "no folder selected" does it happen if you use "Open Folder" ?


Release date for 1.5.10?

We're waiting on https://github.com/fabiolb/fabio/issues/506 for a core functionality we'd like to add to our infrastructure. Is there an estimated date for 1.5.10 or shall we use a custom build for the moment?.
/cc @arrodriguez
Thx for such amazing project.

Just did. Thx for the reminder and the patience ... :)


Binary search for Test-PasswordQuality

Would it be feasible to implement a binary search to improve the comparing speed?
As the password lists from HIBP are pre-sorted, this could be a promising approach.

The search is actually done using a hash table, which contains all AD account password hashes. So the time required to lookup HIBP hashes only increases linearly with the size of the HIBP list. It is therefore faster than a binary search.
Do you have performance issues? Are you sure that they are caused by Test-PasswordQuality itself and not by Get-ADDBAccount/Get-ADReplAccount?
Asking Troy Hunt to publish a reduced HIBP file containing only passwords that meet AD complexity requirements would probably be the way to go. I am also working on replacing regular indices with containerized indices for AD DB account enumeration.

Extracting the hashes using Get-ADDBAccount takes just a few seconds.
The bulk of time is spent using Test-PasswordQuality.
Using the current version pwned-passwords-ntlm-ordered-by-hash-v4.txt (19'891 MBytes)
the Test-PasswordQuality function takes about one hour , regardless of the count of accounts/hashes to test.
Why build a hash table using all 550M hashes from the v4 file and not just for the hashes to be checked, which are  much less (at least for most of us admins out there)?
Using an adapted version of Chris Dent's Binary Searching Script, the searching/comparison time for 2000 hashes against the same v4 file was less than 2 minutes, and this includes the conversion into Hashcat format using Get-ADDBAccount, reading the resulting file line-by-line and writing the results to a new file line-by-line. Here's a link to Chris' Source:
https://gist.github.com/indented-automation/9abd843e752b48ae7910314b878d594b#file-find-string-ps1

Thx for reply, @magl8. Well, the code is doing exactly what you described. It is only storing the hashes to be checked, not those 550M. It only took a couple of minutes on my PC when I tested it the last time. The file is then processed in a single pass, see (TestPasswordQualityCommand.cs)[https://github.com/MichaelGrafnetter/DSInternals/blob/master/Src/DSInternals.PowerShell/Commands/Misc/TestPasswordQualityCommand.cs]:
/// <summary>
/// Maps password hashes to lists of account names.
/// </summary>
private IDictionary<byte[], SortedSet<string>> hashToAccountMap;
...
// We need to cache NT hashes of all accounts for the Duplicate and Weak Password Tests.
this.hashToAccountMap = new Dictionary<byte[], SortedSet<string>>(PasswordDictionaryInitialCapacity, HashEqualityComparer.GetInstance());
...
private void TestWeakPasswordHash(byte[] weakHash)
{
    SortedSet<string> matchingAccounts;
    bool foundAccounts = this.hashToAccountMap.TryGetValue(weakHash, out matchingAccounts);
    if (foundAccounts)
    {
        this.result.WeakPassword.UnionWith(matchingAccounts);
    }
}

So I will have to run a profiler to check where the bottleneck is. It might either be the Dictionary or StreamReader. Or do you have an idea while looking at the code?

Well, I'm not really a programmer...
Looking at your code at a conceptional level, I could imagine that the use of SortedDictionary instead of IDictionary could speed up the searching process (because it inherently uses binary search).
Perhaps it would help to check for the existance of a hash with the ContainsKey method instead of trying to retrieve it in full with the TryGetValue method.

@MichaelGrafnetter:
Thank you for your patience.
Reading up on the subject, I found the problem of your code: You are building your own dictionary hash table, reading the whole 550M perfectly pre-sorted hashes into your own dictionary table. Here's the code snippet:

I suggest you do NOT re-hash the whole pre-sorted file from HIBP. This is unnecessary because the file is already sorted and contains the desired information without the need for conversion. Instead, you can use the HIBP file to directly look up ("cherry-pick") the hash codes taken from the AD extract using a file-based binary searching function.
This results in a searching time of 0.05 seconds per user (using the aforementioned PS code from Chris Dent). I would say that this is a massive improvement compared to a standard 1 hour waiting.
Of course, this approach only works if the pwned ntlm hash list file is pre-sorted by hash.
And looking up more than 72k user hashes at once would be the break even point in comparison to the current approach. For these users and for those requiring the use of unsorted hash lists, you could implement a command line parameter to enforce the building of your own dictionary table.
Do you think it would be feasible to implement a "direct lookup" alternative into DSInternals?

Here is the code I used for the file-based binary search:
function Find-String {
    <#
    .SYNOPSIS
        Find a matching string from an alphabetically sorted file.
    .DESCRIPTION
        Find-String is a specialised binary (half interval) searcher designed to find matches in sorted ASCII encoded text files.
        Usage example: $Pwned = find-string $TestHash pwned-passwords-ntlm-ordered-by-hash-v4.txt
    .NOTES
        Change log:
            11/08/2014 - Chris Dent - First release.
            03/30/2019 - Marius Gaudin - Adaption to haveibeenpwned.com sorted hash list files (both SHA1 and ntlm).
    #>

    [CmdletBinding()]
    [OutputType([Boolean])]
    param (
        # The string to find. The string is treated as a regular expression and must match the beginning of the line.
        [Parameter(Mandatory = $true)]
        [String]$String,

        # The name of the file to search.
        [Parameter(Mandatory = $true)]
        [ValidateScript( { Test-Path $_ } )]
        [String]$FileName
    )

    $FileName = $pscmdlet.GetUnresolvedProviderPathFromPSPath($FileName)
    $FileStream = New-Object IO.FileStream($FileName, [IO.FileMode]::Open)
    $BinaryReader = New-Object IO.BinaryReader($FileStream)
    $pwned=$false

    $Length = $BinaryReader.BaseStream.Length
    $Position = $Length / 2

    [Int64]$HalfInterval = $Length / 2
    $Position = $Length - $HalfInterval

    while ($Position -gt 1 -and $Position -lt $Length -and $Position -ne $LastPosition) {
        $LastPosition = $Position
        $HalfInterval = $HalfInterval / 2

        $BinaryReader.BaseStream.Seek($Position, [IO.SeekOrigin]::Begin) | Out-Null

        # Track back to the start of the line
        while ($true) {
            $Character = $BinaryReader.ReadByte()
            if ($BinaryReader.BaseStream.Position -eq 1) {
                $BinaryReader.BaseStream.Seek(-1, [IO.SeekOrigin]::Current) | Out-Null
                break
            } elseif ($Character -eq [Byte][Char]"`n") {
                break
            } else {
                $BinaryReader.BaseStream.Seek(-2, [IO.SeekOrigin]::Current) | Out-Null
            }
        }

        # Read the line
        $Characters = @()
        if ($BinaryReader.BaseStream.Position -lt $BinaryReader.BaseStream.Length) {
            do {
                $Characters += [Char][Int]$BinaryReader.ReadByte()
            } until ($Characters[-1] -eq [Char]"`n" -or $BinaryReader.BaseStream.Position -eq $BinaryReader.BaseStream.Length)
            $Line = (New-Object String (,[Char[]]$Characters)).split(":")[0].Trim()
         } else {
            # End of file
            $FileStream.Close()
            return $pwned
        }

        if ($Line -match "^$String") {
            # Close the file stream and return the match immediately
            $FileStream.Close()
            $pwned = $true
            return $pwned
        } elseif ($Line -lt $String) {
            $Position = $Position + $HalfInterval
        } elseif ($Line -gt $String) {
            $Position = $Position - $HalfInterval
        }
    }

    # Close the file stream if no matches are found.
    $FileStream.Close()
    return $pwned
}

Thanks, @magl8, for the input. After doing some tests, I have to admit that it really takes too much time to process the file. Even the following simple command took more than 1 hour to finish:
Measure-Command { Get-Content 'pwned-passwords-ntlm-ordered-by-hash-v4.7.txt' }

It would therefore appear that not reading the entire file and doing a binary search is the best option. I will thus add a new parameter called -WeakPasswordHashesOrderedFile and implement the binary search in one of the next DSInternals releases.

Implemented in DSInternals 3.4.


Remove linear map structs and use plain tuples instead.

The changes are intentionally were made close to the original implementation w/o possible simplifications to ease the review
Fixes #63207, supersedes #63379 (and fixes #63234)

Tagging @BradLarson @dan-zheng

@swift-ci please test

Nice approach! This seems to simplify things and avoids the struct complications.
I wonder if there are still complications from the branching trace enums, since those are still NominalTypeDecls who go through various verifications – but that would be separate.

@dan-zheng Yes. I'm afraid we can cook a testcase that would trigger an assertion similar to https://github.com/apple/swift/issues/63207 but due to lack of imported branch trace enum type. Though, I believe it ought to be more complicated.

This is super exciting! For branching trace enums, perhaps we should explore adding a SIL-level sum type construct. Then we may be able to remove SynthesizedFileUnit altogether because it was added just for AD-generated types.

Yeah, right. But maybe as a next step? :)

Sure, this is already strictly an improvement.

@swift-ci please test

@rxwei I'm feeling like opening a can of worms as many things require original AST types, while we'd need to operate mostly on lowered SIL ones. I thought that I fixed all such places, but apparently not. The last failure was caused by assertion inside generic cloner as metatype instruction (which is used when we'd need to allocate linear map context in loopy cases) essentially requires AST type...

@swift-ci please test

@swift-ci please test

@swift-ci please test

MacOS test is broken due to broken main

@swift-ci please test macos

@swift-ci please test macos platform

@swift-ci please test

@swift-ci please test macos platform

@swift-ci please test macos platform


Windows Commands/query: bullet list & technet link

Description:
After noticing the now closed issue ticket #3004 (Which Windows API will be used by this command "query session" "query user") and checking the linked page to see if there is anything obvious that might have answered the ticket question directly for jyxjjj (DESMG-ShenLin), I noticed a couple of small page issues in need of resolving:

The "Additional References" section contains 2 links, but only 1 bullet point (missing list formatting)
The TechNet URL is now permanently redirected to docs.microsoft.com and the redirect page is deprecated, pointing to the 2012 R2 page.

Changes proposed:

Add the missing MarkDown bullet point to the last link
Update the permanently redirected TechNet URL
Use the 2012 R2 page URL instead of the deprecated 2012 page
Replace the "TechNet" reference in the link description with "Microsoft Docs"
Whitespace adjustments:

Add missing blank line after the MarkDown H2/H3 headings (3)
Add 1 blank line between the metadata and the page title

Ticket closure or reference:
Ref. #3004

#sign-off


Refactoring?

It might be simpler to use tables instead of the genome dictionary.

Two tables with all features
If the genome sequence has changed -> include an extra field with the sequence
Get rows that are not present in both tables
Do a similar thing to what is done now

Something similar to what I did in https://github.com/pombase/all_previous_sgd_peptide_sequences


Is it practical to test "unfound" Statements due to voided?

It seems that an LRS may be able to reject a voiding Statement because it doesn't have a Statement Id in the system.  The tests following that may have to adjust.  In other words, this test is only a SHOULD (or SHOULD NOT reject, if you will), even for 1.0.3.

@andyjohnson The part where it gets weird is that the specification states:

A Statement Reference MUST set the "id" property to the UUID of a Statement. There is no requirement for the LRS to validate that the UUID matches a Statement that exists.

When describing a StatementRef, while there is no requirement that I could find around Voided needing any additional requirements.

Hey @DavidTPate thanks!  You are correct.  We changed the closest language in 1.0.1/1.0.2
"Upon receiving a Statement that voids another, the LRS SHOULD return a descriptive error if the target Statement cannot be found."
to
"Upon receiving a Statement that voids another, the LRS SHOULD NOT* reject the request on the grounds of the Object of that voiding Statement not being present."
with the reasoning that the only way an error can be returned realistically is with a rejection.  We don't have any sort of warning in place, even though that was the intent.  We effectively created gray area, which we now seek to undo.
We decided on the call that it would be a good idea to have some of these SHOULD* and SHOULD NOT* behaviors as flags or warnings in the Test Suite.

After discussion, continue rejecting but put warning…


Front-end changes: Gene page

[x] New coloc table where the gene_id is fixed

[x] Title: Colocalising studies
[x] Subtitle: “Which studies have evidence of colocalisation with molecular QTLs for this gene?”

Closing as done


Messagebird's Rebranding to Bird makes Phone Auth Broken

Bug report

[x] I confirm this is a bug with Supabase, not with my own application.
[x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug
The Messagebird integration doesn't work anymore and is broken. I don't receive any SMS codes at all.
To Reproduce
Steps to reproduce the behavior, please provide code snippets or a repository:

Enter Bird credentials
Create an application that uses phone auth.
No SMS gets sent.

Expected behavior
The Bird credentials work with phone auth and the SMS gets received on my phone.
Additional Context
If the supabase team or anyone is open to a PR, please let me know and I would do one if I have time.

Hey @pateljoel,
Thanks for flagging. I think a PR would be welcome if there's a corresponding guide update
It's also possible to use the Send SMS Hook for now.
Let us know if you decide to take it up and if you need any help
Joel


What model is to be used for extraction of the semantic segmentation map?

Thank you for your awesome work released in open source! I really appreciate the impact this paper and code will bring to the community.
I would like to test this model by using in-the-wild video which requires preprocessing.
I am planning to use the follows:

Depth: Depth Anything with greyscale
Dwpose: official dwpose repository
Normal: ICON normal map
mask: unsure whether it is necessary as "inference.yaml" does not require mask for its guidance_types by default

Meanwhile, I am unsure what models I should be using for semantic segmentation map. Please guide me if there is any model that are suitable to be used in the data preprocessing stage.

Also I assume there will be an error for example guide video 7, 8 and 9.
-> it has additional video file "output.mp4" (outpuo.mp4 for video 7)
This goes into "inference.py" and causes an error at line 78 since it is not an image
The error message after replacing the absolute directory using ~ is:
  File “~/miniconda3/envs/champ/lib/python3.10/site-packages/PIL/Image.py", line 3298, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file ‘~/champ/example_data/motions/motion-08/normal/output.mp4'

I assume adding the following lines after line 73 would work?
try: Image.open(guidance_image_path).convert("RGB")
except: continue

Hello, I am trying to get the normal map for my custom data with ICON.
May I ask you to share the code for obtaining normal map from the data?

If you mean the official github repo, here's the link. https://github.com/YuliangXiu/ICON
I would recommend using DSINE instead if you want easier inference here https://github.com/baegwangbin/DSINE
Best wishes :)

If you mean the official github repo, here's the link. https://github.com/YuliangXiu/ICON I would recommend using DSINE instead if you want easier inference here https://github.com/baegwangbin/DSINE Best wishes :)

Thank you for replying! I will try it :)


Prompts example for the demo page

Hi. Thank you very much for this great model.
If I can suggest a little update of the demo page. Can you add one or two samples of the prompts (pos/neg) used to build the demos ?
Thanks a lot !
Greetings from France.

Sorry, just see the prompts list in the github page.


chore: Update pre-commit

Addresses the recent incompatibility between Black & Click

Codecov Report

Merging #1416 (4e2ec4a) into master (4a43cfe) will decrease coverage by 0.03%.
The diff coverage is n/a.

:exclamation: Current head 4e2ec4a differs from pull request most recent head fb62ced. Consider uploading reports for the commit fb62ced to get more accurate results

@@            Coverage Diff             @@
##           master    #1416      +/-   ##
==========================================
- Coverage   99.19%   99.16%   -0.04%
==========================================
  Files          84       84
  Lines        6470     6470
  Branches     1136     1136
==========================================
- Hits         6418     6416       -2
- Misses         26       27       +1
- Partials       26       27       +1

Flag
Coverage Δ

unittests
99.14% <ø> (-0.02%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/schemathesis/cli/callbacks.py
98.33% <0.00%> (-1.67%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a43cfe...fb62ced. Read the comment docs.


pricing: Publish current enterprise pricing

Description

Related Issue(s)

Checklist

[ ] I have read the contribution guidelines
[ ] I have considered the performance impact of these changes
[ ] Suitable unit/system level tests have been added and they pass
[ ] Documentation has been updated

@zackwasli

How can you see the preview of what the suggested changes will look like again?

Since some checks didn't pass, it doesn't create the preview... You could see it if you had it on local.

Netlify appears to now be using node version 20, but the plugin @netlify/plugin-lighthouse says it does not run above version 19.
We'll either need to:

Configure netlify to use an older version of node
Upgrade plugin-lighthouse

The later is going to be fastest, I'll just do that on this PR


[BUG] Application Insights JavaScript(Web) SDK v3 using deprecated APIs affecting lighthouse score

Application Insights JavaScript(Web) SDK v3 using deprecated APIs affecting lighthouse score best practices. Please suggest some way to remove this issue.

As a generic SDK that is designed to run in any web environment we always (by default) attempt to hook all possibly available "unload" handlers. We have a configuration option where you can tell us to not attempt to hook 0 or more events, including the unload event.
Some additional details and a link to the configuration is available in this comment, https://github.com/microsoft/ApplicationInsights-JS/issues/2427#issuecomment-2392203244


Guidance and better defaults for modifying a document during traversal

Hello,
I'm currently writing a static HTML site generator and have settled on Xot after trying several other libraries because it seems to support all the features I need (XHTML support + reasonable efficiency + in-place document modification + rich API for node insertion/modification/removal).
While trying to write a function to remove comment nodes and minifiy text, I just spent a long time confused by unexpected panics and what appeared to be the random removal of sibling nodes I hadn't yet visited and text nodes I could never visit except if I repeatedly traversed the tree.
Representative code:
// Remove comments and outer whitespace from an existing node
fn minify(xot: &mut Xot, node: xot::Node) -> Result<(), xot::Error> {
    if xot.is_comment(node) {
        return xot.remove(node);
    }

    if let Some(text) = xot.text_mut(node) {
        let trimmed = text.get().trim();
        if trimmed != text.get() {
            let trimmed = trimmed.to_string();
            text.set(trimmed);
        }
    }

    let children: Vec<xot::Node> = xot.children(node).collect();
    for child in &children {
        minify(xot, *child)?;
    }

    Ok(())
}

Above, minify is a recursive method that 1) removes comments, 2) strips whitespace off of text nodes and 3) calls itself for all child nodes.
The first issue here is the temporary vector children, which is a simple but slightly inefficient workaround to avoid how the children iterator borrows the Xot, preventing me from mutating the document at the same time. I haven't done much more investigation but I'd love it if there's a canonical way to iterate over children while mutating the document without extra allocation. If there is a recommended way, the documentation for Xot::children would be a great place to describe it. I imagine the following could work but it feels a bit unwieldy:
let mut child = xot.first_child(node);
while let Some(ch) = child {
    // use 'ch' ...
    child = xot.next_sibling(ch);
}

The second issue is that while the code for the minify function feels correct to me, it randomly either panics with internal error: entered unreachable code: Try to access a freed node or I'm left with extra whitespace due to text nodes that mysteriously were never visited. This surprising behaviour turns out to be due to text_consolidation being enabled by default. If I disable it first, the above minify functions works perfectly. I appreciate the intended memory efficiency that text_consolidation offers but it was a nasty surprise and cost me a lot of time debugging what looks to be a perfectly good and simple algorithm. I would suggest either disabling this feature by default, or removing it completely in favour of an explicit method like Xot::consolidate_text(&mut self, node: Xot::Node) for those who actually care about the difference. I expect that for the vast majority of use cases, the surprising negative side effects when doing straightforward recursive DOM manipulation is not worth the memory and complexity savings. Again, clarity and guidance in the documentation would be valuable here.
Otherwise, I'm loving this library and after finding these two workarounds, it's working beautifully for my use case. I'm happy to help prototype things in a pull request if they make sense to you.

Thank you for this feedback! It's interesting to see other people use an API in a way that is reasonable but I took a different approach and never quite thought of doing it this way.
Mutation pattern
I use the following strategy to mutate documents, which does take some extra allocations:

go through the tree immutably and record nodes I want to do something with into a vec. I use something like descendants() for this, or perhaps traverse() if I need structural information and edges.

then afterwards go through those nodes and do the modifications (or deletion) I want.

So I split tree access and tree mutation into separate phases. I suspect that it will have better performance characteristics than doing both in the same phase, but it does have the slight cost of some allocation during the traversal.
Text consolidation
I enable this by default as my style of access rarely presents problems with it, and it results in more reasonable trees. That said, in some cases I do insert a temporary element first (which won't be consolidated) to do mutation.
Your point about unexpected behavior where nodes seem to disappear is an important one. I therefore agree that we should change the default and can change my dependent code to that take into account easily enough.
I also have no objections to adding an explicit method, consolidate_text. Should it consolidate text recursively or only shallowly in a single element?
I don't know whether we should remove automatic text consolidation entirely - it would slightly simplify the code but I don't know whether that would really help much.  In some of my use cases text consolidation is actually mandated by the spec, and I think it often fits user expectations not to be confronted with subsequent text nodes.  When you parse an XML document for instance I think you'd expect consolidated text nodes. So having it maintain that as an invariant can be useful. And see here for instance in case of the XSLT spec:
https://www.w3.org/TR/xslt-30/#constructing-complex-content (point 7)
https://www.w3.org/TR/xslt-30/#constructing-simple-content (point 2)
Interestingly enough I suspect there's a bug where you don't get consolidated text nodes during parsing if you mix a CDATA section with a non-CDATA section, or have two CDATA sections in a row. We should fix that! There are circumstances where rendering as CDATA actually produces such mixed content, for instance here:
https://www.w3.org/TR/xslt-xquery-serialization/#XML_CDATA-SECTION-ELEMENTS
Points of action
I think we can agree on this:

turn off text consolidation by default

add a manual consolidate text method (and figure out what it should do)

ensure that multiple CDATA and mixing CDATA and non-CDATA during parsing results in a single consolidated text node. I don't think text consolidation itself has to be enabled during parsing due to the way parsing actually works.

Document the access/mutate pattern as guidance in the docs somewhere

Document the surprising effect of text node consolidation somewhere

Further examine whether we can get rid of auto-consolidation entirely.

See also #28 about an issue I just realized; if we disable consolidation by default various equality checking code needs to be modified so that it doesn't reject unconsolidated text nodes. (though the fact that it doesn't helped me debug something just now!)


platformio: change name

Apparently, there are two libraries on platformio with the same name "CCS811":
Yours https://platformio.org/lib/show/1609/CCS811
and a other one. https://platformio.org/lib/show/3333/CCS811
If you specify the library by name, it loads the other one. I'm not sure if this is a issue of platformio or your library.

Hi @dadosch,
Uhh, what can I say. I didn't even know I was on Platform IO! I did not put my library there.
The lib is called the same as the sensor name. Sounds like a logical choice. And Adafruit made the same choice. And apparently platformio allows two libs with the same name, that is good. I guess that it then also has a mechanism to select another one if you don't like the default one. I never worked with platformio, so I can't help you here.
I just that a bit of quick clicking in platformio and found something that might help:
# Using library Id
 platformio lib install 1609

# Using library Name
 platformio lib install "CCS811"

# Install specific version
 platformio lib install<EMAIL_ADDRESS>
Thanks for the tip - I feel a bit proud now that the copied my work.

I found your library through platformio, so I actually tried to install your lib with
 platformio lib install "CCS811"
but it installed https://platformio.org/lib/show/3333/CCS811 instead.
Platformio is taking this name from your https://github.com/maarten-pennings/CCS811/blob/master/library.properties file. Adafruit has the following name: name=Adafruit CCS811 Library


[nrf noup] boot: limited pcd routines calls to network core update

this is amendment to
https://github.com/nrfconnect/sdk-mcuboot/commit/d362b129cb18
Signed-off-by: Andrzej Puzdrowski<EMAIL_ADDRESS>fixes>
[57/300] Building C object CMakeFiles/app.dir/main.c.obj
/home/ludu/git/nrfconnect_chip/bootloader/mcuboot/boot/zephyr/main.c: In function 'main':
/home/ludu/git/nrfconnect_chip/bootloader/mcuboot/boot/zephyr/main.c:584:5: warning: implicit declaration of function 'pcd_lock_ram' [-Wimplicit-function-declaration]
584 |     pcd_lock_ram();
|     ^~~~~~~~~~~~
[96/300] Building C object CMakeFiles/app.dir/home/ludu/git/nrfconnect_chip/bootloader/mcuboot/boot/bootutil/src/loader.c.obj
/home/ludu/git/nrfconnect_chip/bootloader/mcuboot/boot/bootutil/src/loader.c: In function 'boot_validated_swap_type':
/home/ludu/git/nrfconnect_chip/bootloader/mcuboot/boot/bootutil/src/loader.c:933:22: warning: implicit declaration of function 'pcd_network_core_update' [-Wimplicit-function-declaration]
933 |             int rc = pcd_network_core_update(vtable, fw_size);
|                      ^~~~~~~~~~~~~~~~~~~~~~~
[293/300] Linking C executable zephyr/zephyr_prebuilt.elf

@hakonfam
CONFIG_NRF53_UPGRADE_NETWORK_CORE=y implies CONFIG_SOC_NRF5340_CPUAPP=y, so maybe worth to reduce preprocessor condition to check on CONFIG_NRF53_UPGRADE_NETWORK_CORE?

CI on nrf manifest PR has passed


Creating cubed arrays from lazy xarray data

To subset my data whilst getting around #196 I tried slicing using xarray's lazy indexing machinery before converting to cubed arrays using .chunk (a trick which when used with dask avoids creating a bunch of open-dataset tasks for chunks that will never be used).
However this causes cubed to throw an error saying that projected blockwise mem exceeds max mem, which I don't understand - I literally haven't asked cubed to do any operations other than creating the array yet.

ValueError: Projected blockwise memory<PHONE_NUMBER>8) exceeds allowed_mem<PHONE_NUMBER>0), including reserved_mem (0)

The size of the projected memory usage also scales with the size of the subset I select.

Actually, maybe that was my fault for putting chunks={}, which didn't read the on-disk chunking as I hoped, instead just returning the whole variable as one big chunk.
Interestingly if I try chunks={'time': 1, 'values': -1} I get a Zarr error from inside the rechunk function
...

File ~/Documents/Work/Code/cubed/cubed/primitive/rechunk.py:35, in rechunk(source, target_chunks, allowed_mem, reserved_mem, target_store, temp_store)
     31 # rechunker doesn't take account of uncompressed and compressed copies of the
     32 # input and output array chunk/selection, so adjust appropriately
     33 rechunker_max_mem = (allowed_mem - reserved_mem) / 4
---> 35 copy_specs, intermediate, target = _setup_rechunk(
     36     source=source,
     37     target_chunks=target_chunks,
     38     max_mem=rechunker_max_mem,
     39     target_store=target_store,
     40     temp_store=temp_store,
     41 )
     43 # source is a Zarr array, so only a single copy spec
     44 if len(copy_specs) != 1:  # pragma: no cover

File ~/Documents/Work/Code/cubed/cubed/vendor/rechunker/api.py:250, in _setup_rechunk(source, target_chunks, max_mem, target_store, target_options, temp_store, temp_options)
    247 # elif isinstance(source, (zarr.core.Array, dask.array.Array)):
    248 elif isinstance(source, zarr.core.Array):
--> 250     copy_spec = _setup_array_rechunk(
    251         source,
    252         target_chunks,
    253         max_mem,
    254         target_store,
    255         target_options=target_options,
    256         temp_store_or_group=temp_store,
    257         temp_options=temp_options,
    258     )
    259     intermediate = copy_spec.intermediate.array
    260     target = copy_spec.write.array

File ~/Documents/Work/Code/cubed/cubed/vendor/rechunker/api.py:296, in _setup_array_rechunk(source_array, target_chunks, max_mem, target_store_or_group, target_options, temp_store_or_group, temp_options, name)
    293     target_chunks = source_chunks
    295 if isinstance(target_chunks, dict):
--> 296     array_dims = _get_dims_from_zarr_array(source_array)
    297     try:
    298         target_chunks = _shape_dict_to_tuple(array_dims, target_chunks)

File ~/Documents/Work/Code/cubed/cubed/vendor/rechunker/api.py:21, in _get_dims_from_zarr_array(z_array)
     18 def _get_dims_from_zarr_array(z_array):
     19     # use Xarray convention
     20     # http://xarray.pydata.org/en/stable/internals.html#zarr-encoding-specification
---> 21     return z_array.attrs["_ARRAY_DIMENSIONS"]

File ~/miniconda3/envs/cubed_xarray/lib/python3.9/site-packages/zarr/attrs.py:74, in Attributes.__getitem__(self, item)
     73 def __getitem__(self, item):
---> 74     return self.asdict()[item]

KeyError: '_ARRAY_DIMENSIONS'

Finally if I try specifying chunks explicitly e.g. chunks={'time': 1} then the indexing works.
However when I actually execute a possibly-related ModuleNotFoundError: No module named 'xarray' is raised by lithops, but the stack trace is not any more informative. I tried adding xarray to the lithops runtime requirements.txt but no change.

I tried adding xarray to the lithops runtime requirements.txt but no change.

I did this then ran
lithops runtime build -f requirements.txt cubed-runtime -b gcp_functions
lithops runtime deploy -b gcp_functions --memory 2048 cubed-runtime

The second line is normally optional since Lithops will auto deploy the first, but I think you may need to run it to force a new deployment. (In the past I have used lithops runtime delete -b gcp_functions cubed-runtime and even lithops clean -b gcp_functions to start again, but I didn't need that this time.)
With that I managed to get the notebook to run.
It has an interesting timeline though:

I think this is due to the large number of rounds in the reduce. This could possibly be improved by trying a larger allowed_mem (since it can then do more aggregations in one go), but there are probably other optimizations that could be done too.

I did this then ran

That's weird - I swear @cisaacstern and I tried exactly the same thing - we definitely did call deploy :sweat_smile: I'll try again but double-check all the commands, and explicitly clean first.
My requirements.txt looked like this:
cubed
lithops[gcf,gcp]
gcsfs
tqdm
xarray == 2023.5.0
cubed-xarray

It has an interesting timeline though:
I think this is due to the large number of rounds in the reduce. This could possibly be improved by trying a larger allowed_mem (since it can then do more aggregations in one go), but there are probably other optimizations that could be done too.

Huh. I'm very pleased that worked at all (real data in a huge store!) but wondering if it's not going to test / demonstrate the kind of scaling that I wanted to try out. I was looking for a problem that would show linear-like weak scaling on a larger and larger dataset, whereas it seems like this would deviate from linear scaling due to the number of aggregation steps.

I think this is due to the large number of rounds in the reduce. This could possibly be improved by trying a larger allowed_mem (since it can then do more aggregations in one go)

I tried running this again with allowed_mem="2GB", but the computation still took about the same time. The last few rounds were smaller, but slower (see below). I need to look into what is happening in more detail, but I think there is the opportunity for more optimisation here.

I changed the mean to a sum and the 21 min runtime went down to 7 min. I think this shows that the overhead of using structured arrays is significant, and it would be worth implementing #69.

Closing as this is fixed now


Build customizable Navigation Bar Component

Build a customizable Navigation Bar Component. It should be fully responsive.

Solved by #9


vuestorefront is redirecting to alokai.com

https://www.vuestorefront.io should be changed to https://alokai.com/

I am interested to work on this issue


Simple command line tool for browsing schema endpoints

This adds a quick and dirty command line tool for browsing through the endpoints in a schema.
Arrow keys allow selection of an endpoint, [Q] to exit.
Usage:
$ python3 output/viewer.py output/schema/schema.json

Screenshot:

Sweet! Could you please add the command as target: make spec-inspect to the Makefile?

Indeed I can.