feat: Added api, error, page boilerplate

#__
Checklist

[ ] I did't over-scoped my PR
[ ] My PR title match the commit convention
[ ] I did not include breaking change
[ ] I included unit test that cover my changes
[ ] I added/updated the documentation about my changes
[ ] I made my own code-review before requesting one

Description of the changes

Technical highlight/advice

:tada: This PR is included in version 1.0.0-alpha.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Add benchmarking

Add some way to easily generate benchmark results.

It's not proper benchmarking of course, but, the TestCLI workflow output captures timing information that can be a useful starting point, e.g. my 2018 day 5 solution probabaly could use some optimization! https://github.com/pbouffard/AoCElfTools.jl/actions/runs/3785242841/jobs/6435147205#step:7:29


Use port 8080 everywhere, add comments to .env.example

Update PORT to 8080 everywhere.  Also added some comments to .env.example while I was in there.  Prettier reformatted the CONTRIBUTING.md doc as well.

I should put #122 in first


Color Accessibility Issue

Message from a Twitter Follower:

@jwunderl
The only challenge is hexcode -> named color, as the color palette can be changed in the project. Should be very easy enough to just do a quick switch case for the default colors we have though (we already have Color {n} as the tooltip, would just be appending or replacing the name into that).


Speed up single static virtual satellite extraction

Currently we are slower for static satellite extraction from multiple frames than Analysator, because we store the sorted CellIDs as Dict for each timestep.
200 files on Mahti, serial processing:

Julia 1.8
Python 3.9

real time [s]
3m21.223
1m5.901

We may need to speed up this task.

Now we implement a specific function extractsat:
300 files on Mahti, serial processing:

Julia 1.8
Python 3.9

extraction time [s]
8.46
92.79

More than 10x faster!


Corrupt styling when editing text

Description

If you create some text 👎
Outside there is a block paved garden to the front and to the rear is the yard.
Entrance Hall
Then, highlight "Entrance Hall", make it bold
Place your cursor before the E and press delete until your cursor is at the "yard."
Press the </> button, there is a spurious:
<b style="font-size: 1.6rem;">Entrance Hall</b>
This has got to be a bug????

Cannot reproduce anymore :)


Drop default suffix.

This test assumes that the cluster is configured with .svc.cluster.local, which isn't always the case.  Downstream I'm testing against KinD with a randomized suffix and this blew up.
The simplest fix is to drop the .cluster.local and rely on the resolution to find the right thing.
/kind bug
Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:

[ ] Includes tests (if functionality changed/added)
[ ] Includes docs (if user facing)
[ ] Commit messages follow commit message best practices
[ ] Release notes block has been filled in or deleted (only if no user facing changes)

See the contribution guide for more details.
Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes
If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.
Release Notes
NONE

/assign @ImJasonH @vdemeester @afrittoli

Get "https://registry.arendelle-wlx9x.svc:5000/v2/": http: server gave HTTP response to HTTPS client

The saga continues...

Ah, without the .local I think GGCR is expecting TLS....  (sigh).


utm to wgs84 ,Unserializable

I used this jar（proj4j-0.1.0.jar）, but many classes can't be serialized.So I have to modify the source code。。。

@wangqianjia this seems unrelated to the Front End of Search Maven Org, but with the actual use of the library proj4j.
I would suggest to do further research and/or get in touch on github with them here https://github.com/locationtech/proj4j


Display of README is strange

This may not be within our control @mtshiba

Perhaps the previous  tag is not closed.


ros data to xviz

Is it possible converting ros x.bag data to xviz?
if it is possible how can i do it?

Now that @twojtasz has his account back I am closing this as a duplicate of #372.  That covers an offline conversion tool/library.

@twojtasz possible to share any update? It would be great for our project. If you are busy it would be great to get some pointers to your approach and I'm happy complete and contribute it back.

@twojtasz is making progress building out the server code infrastructure:

#435 - the IO module
#453 - the server module

Next up is a formal version of his ROS code.


Add middleware for echo

some metal-toolbox/infratographer projects are adopting echo, we should cater for them and provide audit middleware for them to use.

Closed by #155


onChange in codeview mode

Hello, the onChange method should be called when we change the the text while in codeview mode and atm the onChange trigger only when we switch back to normal view.

Sorry to dig this up because I have the same problem like you did, so can I ask if you have any solution?

On submit of the form I do disable it and toggle the codeview and on the next componentDidUpdade onChange is already called and we have what we need.


tp bug

Hello,
Plugin is amazing BUT whenever I teleport players to a specific location they are looking at South??

Ah whoops found the problem. Should have a fix out today or tommorow.

Ah whoops found the problem. Should have a fix out today or tommorow.

Thank you :)

Fixed with latest update.


Add parsing of dim_param onnx field in driver

More common in newer models such as Whisper, Llama, etc

Ideas on how to implement a basic case where dim_param is used:
Under  src/onnx/onnx_parser.cpp:parse_type:
We need to change this so that we can also get the dim_param() name if it exists. Then pass in a map to parse_type() that can set the value of the dimension based on dim_param name.
Since this function belongs to the onnx_parser class, we can just have the map as a member variable, like it's used for map_input_dims already. Then follow similar logic in the driver source code for --input-dim.

There is a function has_dim_param() defined by protobuf which we can use to implement the check for dim_params.


Add transformers recognizer sample

Change Description
Describe your changes
Issue reference
This PR fixes issue #XX
Checklist

[x] I have reviewed the contribution guidelines
[x] I have signed the CLA
[ ] My code includes unit tests
[ ] All unit tests and lint checks pass locally
[x] My PR contains documentation updates / additions if required

@omri374 How can this be used with other NER models from huggingface? How could a model be specified?

Thanks, Omri!
On Tue, Oct 18, 2022 at 3:03 PM Omri Mendels @.***>
wrote:

Hey @shamikbose https://github.com/shamikbose, here's an example
already on Huggingface Hub: https://huggingface.co/spaces/omri374/presidio,
and specifically this file:
https://huggingface.co/spaces/omri374/presidio/blob/main/transformers_recognizer.py
There's a model_path parameter which lets you specify a huggingface
model. For example:
from presidio_analyzer import AnalyzerEngine, RecognizerRegistrytransformers_recognizer = TransformersRecognizer(model_path="obi/deid_roberta_i2b2")
registry = RecognizerRegistry()registry.add_recognizer(transformers_recognizer)
analyzer = AnalyzerEngine(registry=registry)
results = analyzer.analyze(
text="My name is Christopher and I live in Irbid.",
language="en",
return_decision_process=True,
)for result in results:
print(result)
print(result.analysis_explanation)
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/presidio/pull/866#issuecomment-1282874871,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMD3OJMP67KDZ3OA5WCHPWTWD3X7LANCNFSM5WNB2CMA
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
-Regards,  Shamik Bose


Settings App Docs Missing Two URIs

This is some great info, and has been very useful. However, two page URIs seem to be missing, both in the Ease of Access category:

Cursor and Pointer Size
Color Filters

Here's the page URL: https://docs.microsoft.com/en-us/windows/uwp/launch-resume/launch-settings-app#user--accounts

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c6a2ae18-080e-3470-d1c7-ab2990d4da85
Version Independent ID: 3d0d02af-c1d7-e2e7-ba01-15e0904dc9a6
Content: Launch the Windows Settings app - UWP app developer
Content Source: windows-apps-src/launch-resume/launch-settings-app.md
Service: unspecified
GitHub Login: @TylerMSFT
Microsoft Alias: twhitney

Both URIs are in the documentation and working, this issue can be closed.

Confirmed, they are in the docs. Closing.


enhance: disconnect player, switch player, decoration, click-through, e.g.

Current solution, that via UNIX signals, is generally a bad idea due to it's potential complexity and bad integration with all WM/DE.
todo: set a ComboBox for player switch interface, and

[ ] tray icon
[ ] dock actions

these integration needs D-Bus service registeration

another option: right click menu on decoration


Bower dependancies angular version

Hello,
I'm using Angular 1.4.7 and since your last commit, I can't install my bower dependancies because of the new version of angular required.
It's possible to publish a tag for the previous version of angular ?
I can't upgrade my Angular version on my project.
Thanks.

Hello,
I've made two releases (with and without the package update). So you can use the previous one in your project : [tag 0.1.2 (https://github.com/Wlada/angular-carousel-3d/releases/tag/0.1.2).
Regards,
Luwangel


Generate load test scripts from REST API specs

We have an effort going on to generate JMeter and LoadRunner code from specs in Swagger, RAML, WADL, etc. It would be useful to have DSL generated for Gatling too? As I describe in
See http://github.com/wilsonmar/oss-perf  it is doable now that REST API specs are designed to be complete.  This would be a major revolution because devs won't have to wait for test scripts to be hacked together.

Seems someone asked for the same thing on the Swagger side and answer was "contributions welcome".
Flagging the same way here.


Implement author selection and assign action

SS

The author names should be clickable, similar to the author search
author selection and assign

Create AssignAuthorSearchDrawer similar to ExistingConferencesDrawer

needs new namespace
should display a simple search box (antd/Input.Search), that will set q query for search (
renderItem function should render radio next to the each result
Assign should use API endpoint created #1092

where from_author_recid is recid of the profile, to_author_recid is recid of from radio button selection, literature_recids is selectionMap.fitler((key, value) => value)).map((key, value) => key) (see #1100 for how to access selectionMap)
should use antd/PopConfirm before calling the API
should close the drawer and clear the selection (see #1100) when successful
New Author is selected, request should be done without to_author_recid . If new author is selected, the profile of this new author should open in a new tab

Separated from #1095


Does SparkOperator support rebooting stateful spark jobs?

Does SparkOperator support rebooting stateful spark jobs? And is it possible to just use spark-submit and k8s Deployments to submit and refresh those spark jobs?

What you meant by rebooting? Did you mean restarting an application on failures for example for streaming applications? If so, yes, the operator allows you to configure a RestartPolicy.

What you meant by rebooting? Did you mean restarting an application on failures for example for streaming applications? If so, yes, the operator allows you to configure a RestartPolicy.

Nope, not on failures. I mean can we proactively update the spark job image and reboot the spark job while still keeping the checkpoint data from the previous spark job? Thanks @liyinan926

By proactively update the spark job image did you mean updating the container image of your job?

By proactively update the spark job image did you mean updating the container image of your job?

Yes

We do support restart on spec updates. So if you update the image of your SparkApplication spec, the operator will try to kill the current run and re-start it. Checkpointing is up to your application.

We do support restart on spec updates. So if you update the image of your SparkApplication spec, the operator will try to kill the current run and re-start it. Checkpointing is up to your application.

Thanks @liyinan926 ! Just to confirm, I can also use the the command below to restart the spark jobs (deleting the spark executors not the driver):
kubectl -n <namespace> delete pods -l spark-job-app=<app-name>
However, this won't keep the checkpointing data, right?
While using the SparkApplication in the Operator, it will use the previous checkpointing data, right?

How you restart a job with a different image without deleting and restarting your driver pod? Checkpointing is totally up to the applications in Spark on k8s. The k8s scheduler backend is not involved in checkpointing at all.

Can I know more about Checkpointing is totally up to the applications in Spark on k8s.? When we delete the spark job pods, are these checkpointing data will be deleted as well? Is there a way to preserve these data while updating their images and restart them?

It all depends on where your checkpoint data is stored. If it's stored on HDFS, yes, the data is preserved even if the driver pod is restarted. If the checkpoint data is stored on some persistent volume, again, the data is preserved in case of driver restart. I assume you were talking about Spark streaming checkpoint data, right?

Can I know more about Checkpointing is totally up to the applications in Spark on k8s.? When we delete the spark job pods, are these checkpointing data will be deleted as well? Is there a way to preserve these data while updating their images and restart them?

It all depends on where your checkpoint data is stored. If it's stored on HDFS, yes, the data is preserved even if the driver pod is restarted. If the checkpoint data is stored on some persistent volume, again, the data is preserved in case of driver restart. I assume you were talking about Spark streaming checkpoint data, right?

Yes. Ah, I see your point. I guess we need to figure out a way to handle the data persistence by ourselves (will try HDFS). BTW, last question, I read the document on https://spark.apache.org/docs/latest/running-on-kubernetes.html and it seems like the official spark-k8s only supports spark-submit command to submit spark jobs. And there's no easy way to expand/shrink capacity or even stop the existing pods.  While using the Operators, we can easily achieve these, right? And are the Operators also stop the driver pod each time it tries to expand/shrink the capacity of the spark job pods? Thanks!

Yes. Ah, I see your point. I guess we need to figure out a way to handle the data persistence by ourselves (will try HDFS). BTW, last question, I read the document on https://spark.apache.org/docs/latest/running-on-kubernetes.html and it seems like the official spark-k8s only supports spark-submit command to submit spark jobs. And there's no easy way to expand/shrink capacity or even stop the existing pods.  While using the Operators, we can easily achieve these, right? And are the Operators also stop the driver pod each time it tries to expand/shrink the capacity of the spark job pods? Thanks!

Yes, the operator gives you a way to declaratively specify your job and is able to handle lifecycle management for your job. When you use the operator, the life time of your job is tied to your SparkApplication resource. When you scale your job up/down to use more/less executors, the operator will handles job restart for you.

Yes. Ah, I see your point. I guess we need to figure out a way to handle the data persistence by ourselves (will try HDFS). BTW, last question, I read the document on https://spark.apache.org/docs/latest/running-on-kubernetes.html and it seems like the official spark-k8s only supports spark-submit command to submit spark jobs. And there's no easy way to expand/shrink capacity or even stop the existing pods.  While using the Operators, we can easily achieve these, right? And are the Operators also stop the driver pod each time it tries to expand/shrink the capacity of the spark job pods? Thanks!

Yes, the operator gives you a way to declaratively specify your job and is able to handle lifecycle management for your job. When you use the operator, the life time of your job is tied to your SparkApplication resource. When you scale your job up/down to use more/less executors, the operator will handles job restart for you.

Great, thanks so much!!!

Closing this. Let us know if you have any questions through the Slack channel at https://kubernetes.slack.com/messages/CALBDHMTL.


Improve documentation on creating a new user from Golang

TLDR: Request to add a code snippet about adding new Auth record for creating IE creating a user to the documentation.
I was writing tests and I wanted my test app to create a new test user each time I ran test pragmatically rather than having to go into the UI to do it.
I read through all the documentation and spent lots of time reading the source code until I finally came up with.
 // add a test user
    collection, err := dao.FindCollectionByNameOrId("users")
    if err != nil {
        panic("error retreeiving users collection, " + err.Error())
    }
    record := models.NewRecord(collection)

    form := forms.NewRecordUpsert(testApp, record)
    form.Email = TestUserEmail
    form.Password = TestUserPwd
    form.PasswordConfirm = TestUserPwd

    if err := form.Submit(); err != nil {
        panic("form failed to submit +" + err.Error())
    }

This seems to work for my purposes of adding a test user to the testApp when I spin it up.
It would be nice if we could mark this in the docs since this took me too long to find out how to do (subjective, took about an hour,). I got a little confused since  in the db we just store the passwordHash, so i was trying to find out if I need to hash it myself when adding the record via go, or there is a thing like above that does the hashing for me (which is much better)
Great work with pocketbase! I think its the best out of the box setup for using SQLITE out there I am loving it so far,  especially the integration tests api!
Keep it up!

It won't be needed. This is simplified with the refactoring and in the next upcoming v0.23 release (there are no ETAs yet) all system fields (id, password, email, etc.) are regular collection fields and you can create auth records just by doing:
record := models.NewRecord(collection)
record.Set("password", "1234567890")    // or the designated helper SetPassword
record.Set("email"<EMAIL_ADDRESS>// or the designated helper SetEmail
app.Save(record) // or app.SaveNoValidate(record) to skip fields validation

At the moment creating an auth record using the RecordUpsert is correct when you want validation. If you want a direct DB insert you have to save the record using the app.Dao() as shown in https://pocketbase.io/docs/go-migrations/#creating-new-auth-record.


Support for OpenAI completion models, in particular gpt-3.5-turbo-instruct

Released today - no OpenAI blog post yet: https://news.ycombinator.com/item?id=37558911

Using this model involves openai.Completion.create as opposed to openai.ChatCompletion.create`.
API documentation: https://platform.openai.com/docs/api-reference/completions/create

Got this prototype working:
diff --git a/llm/default_plugins/openai_models.py b/llm/default_plugins/openai_models.py
index fd8c689..8b7854e 100644
--- a/llm/default_plugins/openai_models.py
+++ b/llm/default_plugins/openai_models.py
@@ -22,6 +22,10 @@ def register_models(register):
     register(Chat("gpt-3.5-turbo-16k"), aliases=("chatgpt-16k", "3.5-16k"))
     register(Chat("gpt-4"), aliases=("4", "gpt4"))
     register(Chat("gpt-4-32k"), aliases=("4-32k",))
+    register(
+        Completion("gpt-3.5-turbo-instruct"),
+        aliases=("3.5-instruct", "chatgpt-instruct"),
+    )
     # Load extra models
     extra_path = llm.user_dir() / "extra-openai-models.yaml"
     if not extra_path.exists():
@@ -249,6 +253,32 @@ class Chat(Model):
             messages.append({"role": "system", "content": prompt.system})
         messages.append({"role": "user", "content": prompt.prompt})
         response._prompt_json = {"messages": messages}
+        kwargs = self.build_kwargs(prompt)
+        if stream:
+            completion = openai.ChatCompletion.create(
+                model=self.model_name or self.model_id,
+                messages=messages,
+                stream=True,
+                **kwargs,
+            )
+            chunks = []
+            for chunk in completion:
+                chunks.append(chunk)
+                content = chunk["choices"][0].get("delta", {}).get("content")
+                if content is not None:
+                    yield content
+            response.response_json = combine_chunks(chunks)
+        else:
+            completion = openai.ChatCompletion.create(
+                model=self.model_name or self.model_id,
+                messages=messages,
+                stream=False,
+                **kwargs,
+            )
+            response.response_json = completion.to_dict_recursive()
+            yield completion.choices[0].message.content
+
+    def build_kwargs(self, prompt):
         kwargs = dict(not_nulls(prompt.options))
         if self.api_base:
             kwargs["api_base"] = self.api_base
@@ -267,29 +297,45 @@ class Chat(Model):
             kwargs["api_key"] = "DUMMY_KEY"
         if self.headers:
             kwargs["headers"] = self.headers
+        return kwargs
+
+
+class Completion(Chat):
+    def __str__(self):
+        return "OpenAI Completion: {}".format(self.model_id)
+
+    def execute(self, prompt, stream, response, conversation=None):
+        messages = []
+        if conversation is not None:
+            for prev_response in conversation.responses:
+                messages.append(prev_response.prompt.prompt)
+                messages.append(prev_response.text())
+        messages.append(prompt.prompt)
+        response._prompt_json = {"messages": messages}
+        kwargs = self.build_kwargs(prompt)
         if stream:
-            completion = openai.ChatCompletion.create(
+            completion = openai.Completion.create(
                 model=self.model_name or self.model_id,
-                messages=messages,
+                prompt="\n".join(messages),
                 stream=True,
                 **kwargs,
             )
             chunks = []
             for chunk in completion:
                 chunks.append(chunk)
-                content = chunk["choices"][0].get("delta", {}).get("content")
+                content = chunk["choices"][0].get("text") or ""
                 if content is not None:
                     yield content
             response.response_json = combine_chunks(chunks)
         else:
-            completion = openai.ChatCompletion.create(
+            completion = openai.Completion.create(
                 model=self.model_name or self.model_id,
-                messages=messages,
+                prompt="\n".join(messages),
                 stream=False,
                 **kwargs,
             )
             response.response_json = completion.to_dict_recursive()
-            yield completion.choices[0].message.content
+            yield completion.choices[0]["text"]

 def not_nulls(data) -> dict:
@@ -303,6 +349,9 @@ def combine_chunks(chunks: List[dict]) -> dict:

     for item in chunks:
         for choice in item["choices"]:
+            if "text" in choice and "delta" not in choice:
+                content += choice["text"]
+                continue
             if "role" in choice["delta"]:
                 role = choice["delta"]["role"]
             if "content" in choice["delta"]:

Example usage:
llm -m chatgpt-instruct 'A poem about otters:'

Graceful and sleek, through the water they glide
With playful spirits and

llm -m chatgpt-instruct 'A poem about otters:' -o max_tokens 128

Graceful creatures of the sea
With fur so soft and eyes so keen
Otters, playful, wild and free
Inhabitants of a watery scene

With sleek bodies and flippers strong
They glide through water with ease
Their movements seem to flow along
As if dancing to the ocean’s breeze

Their playful nature knows no bounds
As they frolic and play all day
In search of fish, they can be found
In a game of hide and seek they play

Their laughter echoes through the waves
A joyful sound for all to hear
In their underwater caves
They have nothing to fear

Got a working version in. Still needed:

[ ] Am I going to increase the default max_token setting?
[ ] A mechanism for registering other completion models
[ ] Documentation

One option for system prompts: chuck it in **bold** at the start of the message.

That default token size of 16 is really small.
https://platform.openai.com/playground?mode=complete defaults to 256 so I'm going to use that default instead.

This is annoying: with a default max_tokens of 256 it seems to just cut off half way through the output:
llm -m instruct 'poem about an otter'

Graceful and sleek, the otter glides
Through the water, a creature of tides
With fur so soft and eyes so bright
It's hard not to be captivated by its sight

In streams and rivers, it loves to play
Diving, twisting, in a carefree way
It's an expert swimmer, with webbed feet
And a long tail, it uses to steer and greet

Its diet consists of fish and shellfish too
Crunching and munching, with a satisfied chew
But it's not just about survival, for this creature
It enjoys each meal, like it's a special feature

With a playful nature and curious mind
The otter explores, never one to find
A boring moment, in its watery home
It loves to adventure and freely roam

Cuddling with its family, on a cozy bed
The otter rests its tired head
A protector, provider, a loving mate
In its own way, it shows its love and fate

A symbol of joy and adaptability
The otter teaches us, with its own ability
To find happiness, in the simple things
And the importance of family and the joy it brings

Oh otter, with your

I checked and the output there is exactly 256. I was hoping it would be somehow aware of that limit.

I'm going to add a logprobs option: https://platform.openai.com/docs/api-reference/completions/create#completions/create-logprobs

integer or null, Optional, Defaults to null
Include the log probabilities on the logprobs most likely tokens, as well the chosen tokens. For example, if logprobs is 5, the API will return a list of the 5 most likely tokens. The API will always return the logprob of the sampled token, so there may be up to logprobs+1 elements in the response.
The maximum value for logprobs is 5.

Tried this:
llm -m instruct 'poem about an otter' -o max_tokens 10 -o logprobs 3

Looks like i need to take extra steps to get it to show up in the logged database response.

I tried storing the full recursive dictionary version of the response if logprobs are present, ended up with this:
sqlite-utils "$(llm logs path)" 'select * from responses order by id desc limit 1' | jq '.[0].response_json' -r | jq

{
  "content": "\n\nIn the river, sleek and sly,\n",
  "role": null,
  "finish_reason": null,
  "chunks": [
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": "\n\n",
          "index": 0,
          "logprobs": {
            "tokens": [
              "\n\n"
            ],
            "token_logprobs": [
              -0.19434144
            ],
            "top_logprobs": [
              {
                "\n\n": -0.19434144,
                "\n": -2.2880914,
                " \n\n": -5.4443407
              }
            ],
            "text_offset": [
              19
            ]
          },
          "finish_reason": null
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": "In",
          "index": 0,
          "logprobs": {
            "tokens": [
              "In"
            ],
            "token_logprobs": [
              -1.7583014
            ],
            "top_logprobs": [
              {
                "In": -1.7583014,
                "S": -1.3833013,
                "Grace": -1.7426763
              }
            ],
            "text_offset": [
              21
            ]
          },
          "finish_reason": null
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": " the",
          "index": 0,
          "logprobs": {
            "tokens": [
              " the"
            ],
            "token_logprobs": [
              -0.13828558
            ],
            "top_logprobs": [
              {
                " the": -0.13828558,
                " a": -2.6539104,
                " rivers": -3.9507854
              }
            ],
            "text_offset": [
              23
            ]
          },
          "finish_reason": null
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": " river",
          "index": 0,
          "logprobs": {
            "tokens": [
              " river"
            ],
            "token_logprobs": [
              -0.78688633
            ],
            "top_logprobs": [
              {
                " river": -0.78688633,
                " water": -3.0681362,
                " sparkling": -3.099386
              }
            ],
            "text_offset": [
              27
            ]
          },
          "finish_reason": null
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": ",",
          "index": 0,
          "logprobs": {
            "tokens": [
              ","
            ],
            "token_logprobs": [
              -0.94506526
            ],
            "top_logprobs": [
              {
                ",": -0.94506526,
                "'s": -1.0700654,
                " she": -2.6325655
              }
            ],
            "text_offset": [
              33
            ]
          },
          "finish_reason": null
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": " sleek",
          "index": 0,
          "logprobs": {
            "tokens": [
              " sleek"
            ],
            "token_logprobs": [
              -1.239402
            ],
            "top_logprobs": [
              {
                " sleek": -1.239402,
                " swift": -0.723777,
                " graceful": -3.2550268
              }
            ],
            "text_offset": [
              34
            ]
          },
          "finish_reason": null
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": " and",
          "index": 0,
          "logprobs": {
            "tokens": [
              " and"
            ],
            "token_logprobs": [
              -0.00032777296
            ],
            "top_logprobs": [
              {
                " and": -0.00032777296,
                " as": -8.484702,
                " ot": -10.187827
              }
            ],
            "text_offset": [
              40
            ]
          },
          "finish_reason": null
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": " s",
          "index": 0,
          "logprobs": {
            "tokens": [
              " s"
            ],
            "token_logprobs": [
              -1.6860211
            ],
            "top_logprobs": [
              {
                " s": -1.6860211,
                " swift": -1.1078961,
                " quick": -1.7485211
              }
            ],
            "text_offset": [
              44
            ]
          },
          "finish_reason": null
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": "ly",
          "index": 0,
          "logprobs": {
            "tokens": [
              "ly"
            ],
            "token_logprobs": [
              -0.014393331
            ],
            "top_logprobs": [
              {
                "ly": -0.014393331,
                "velte": -4.3425183,
                "li": -7.6550174
              }
            ],
            "text_offset": [
              46
            ]
          },
          "finish_reason": null
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": ",\n",
          "index": 0,
          "logprobs": {
            "tokens": [
              ",\n"
            ],
            "token_logprobs": [
              -0.79212964
            ],
            "top_logprobs": [
              {
                ",\n": -0.79212964,
                "\n": -0.7452547,
                "  \n": -3.3858798
              }
            ],
            "text_offset": [
              48
            ]
          },
          "finish_reason": "length"
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    },
    {
      "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
      "object": "text_completion",
      "created":<PHONE_NUMBER>,
      "choices": [
        {
          "text": "",
          "index": 0,
          "logprobs": {
            "tokens": [],
            "token_logprobs": [],
            "top_logprobs": [],
            "text_offset": []
          },
          "finish_reason": "length"
        }
      ],
      "model": "gpt-3.5-turbo-instruct"
    }
  ],
  "id": "cmpl-80LVJQkySEtmJWWug1AH0t7IN8vOO",
  "object": "text_completion",
  "model": "gpt-3.5-turbo-instruct",
  "created":<PHONE_NUMBER>
}

That's pretty noisy though! Imagine that for a much longer response.

I'll try storing a logprobs list of {"text": "...", "top_logprobs": {...}} instead.

That's a bit neater:
llm -m instruct 'slogan for an an otter-run bakery' -o max_tokens 16 -o logprobs 3

"Fresh Treats That'll Make You Otterly Happy!"

sqlite-utils "$(llm logs path)" 'select * from responses order by id desc limit 1' \
  | jq '.[0].response_json' -r | jq

{
  "content": "\n\n\"Fresh Treats That'll Make You Otterly Happy!\"",
  "role": null,
  "finish_reason": null,
  "logprobs": [
    {
      "text": "\n",
      "top_logprobs": [
        {
          "\n": -1.9688251,
          "\n\n": -0.3125751,
          ":": -3.968825
        }
      ]
    },
    {
      "text": "\n",
      "top_logprobs": [
        {
          "\n": -0.33703142,
          "\"": -1.5089064,
          "\"S": -3.8995314
        }
      ]
    },
    {
      "text": "\"",
      "top_logprobs": [
        {
          "\"": -0.13510895,
          "\"S": -2.588234,
          "\"B": -4.119484
        }
      ]
    },
    {
      "text": "Fresh",
      "top_logprobs": [
        {
          "Fresh": -1.4623433,
          "Where": -1.1498433,
          "Ind": -2.5560932
        }
      ]
    },
    {
      "text": " Treat",
      "top_logprobs": [
        {
          "ly": -0.39274898,
          " treats": -1.861499,
          " baked": -3.4396236
        }
      ]
    },
    {
      "text": "s",
      "top_logprobs": [
        {
          "s": -6.5092986e-06,
          ",": -13.406255,
          "z": -13.82813
        }
      ]
    },
    {
      "text": " That",
      "top_logprobs": [
        {
          " from": -0.78117555,
          " Straight": -1.6874255,
          ",": -1.7968005
        }
      ]
    },
    {
      "text": "'ll",
      "top_logprobs": [
        {
          "'ll": -1.1900537,
          " Will": -1.4088038,
          " Make": -1.4556787
        }
      ]
    },
    {
      "text": " Make",
      "top_logprobs": [
        {
          " Make": -0.14272869,
          " Have": -2.8614783,
          " Leave": -3.5802286
        }
      ]
    },
    {
      "text": " You",
      "top_logprobs": [
        {
          " You": -0.26677564,
          " Your": -1.6730256,
          " a": -3.1574006
        }
      ]
    },
    {
      "text": " Ot",
      "top_logprobs": [
        {
          " Ot": -0.58028656,
          " Flip": -2.2677865,
          " Float": -2.2990365
        }
      ]
    },
    {
      "text": "ter",
      "top_logprobs": [
        {
          "ter": -0.0004432111,
          "term": -8.4223175,
          "terr": -9.3598175
        }
      ]
    },
    {
      "text": "ly",
      "top_logprobs": [
        {
          "ly": -0.040374786,
          "-": -3.7903748,
          "-L": -4.8216248
        }
      ]
    },
    {
      "text": " Happy",
      "top_logprobs": [
        {
          " Happy": -0.18345116,
          " S": -1.9959509,
          " Del": -3.8553257
        }
      ]
    },
    {
      "text": "!\"",
      "top_logprobs": [
        {
          "!\"": -0.035835575,
          "\"": -3.4264605,
          "!\"\n": -7.2077103
        }
      ]
    },
    {
      "text": "",
      "top_logprobs": []
    }
  ],
  "id": "cmpl-80LaT4oS6rQPFUg3BfbNn7xBsoIix",
  "object": "text_completion",
  "model": "gpt-3.5-turbo-instruct",
  "created":<PHONE_NUMBER>
}

It's a bit annoying that the only way to see the log probs is to dig around in the SQLite database for them.
I can't think of a clean way to let people opt into seeing them on stderr or similar though.
One option would be to teach the llm logs Markdown output how to display them. That's nicer than messing around in SQLite directly.

Since I opened a fresh issue for it I won't consider logprobs display any more here.

Suffix support is unique to completion models and looks interesting too: https://platform.openai.com/docs/api-reference/completions/create#completions/create-suffix

In trying to write a test for the new logprobs stuff I dropped this into the top of cli.py:
import requests
import json

def log_json(response, *args, **kwargs):
    try:
        data = response.json()
        print(json.dumps(data, indent=4))
    except ValueError:
        # No JSON data in the response
        pass
    return response

import openai
openai.requestssession = requests.Session()
openai.requestssession.hooks['response'].append(log_json)

This worked, but only with --no-stream (since otherwise the event source stuff broke the JSON parsing). I got back this:
llm -m instruct 'slogan for an an otter-run bakery' -o max_tokens 16 -o logprobs 3 --no-stream

{
    "id": "cmpl-80M2iXLmVOtxrxosK1crpgoxvpk2x",
    "object": "text_completion",
    "created":<PHONE_NUMBER>,
    "model": "gpt-3.5-turbo-instruct",
    "choices": [
        {
            "text": "\n\n\"Fresh treats straight from the riverbank!\"",
            "index": 0,
            "logprobs": {
                "tokens": [
                    "\n\n",
                    "\"",
                    "Fresh",
                    " treats",
                    " straight",
                    " from",
                    " the",
                    " river",
                    "bank",
                    "!\""
                ],
                "token_logprobs": [
                    -0.3125751,
                    -0.18438435,
                    -1.3464811,
                    -1.6608365,
                    -2.354677,
                    -0.0038274676,
                    -0.013088844,
                    -0.15179907,
                    -1.417837,
                    -0.41310275
                ],
                "top_logprobs": [
                    {
                        "\n\n": -0.3125751,
                        "\n": -1.9688251,
                        ":": -3.968825
                    },
                    {
                        "\"": -0.18438435,
                        "\"S": -2.293759,
                        "\"B": -3.9812593
                    },
                    {
                        "Fresh": -1.3464811,
                        "Where": -1.1277312,
                        "Making": -2.549606
                    },
                    {
                        " treats": -1.6608365,
                        "ly": -0.48896152,
                        " baked": -3.2389612
                    },
                    {
                        " from": -1.0734268,
                        ",": -1.276552,
                        " that": -2.167177
                    },
                    {
                        " from": -0.0038274676,
                        " out": -5.738202,
                        " ot": -7.519452
                    },
                    {
                        " the": -0.013088844,
                        " our": -4.4662137,
                        " nature": -6.825588
                    },
                    {
                        " river": -0.15179907,
                        " ot": -2.308049,
                        " water": -3.917424
                    },
                    {
                        "bank": -1.417837,
                        "'s": -0.5584622,
                        " bank": -2.917837
                    },
                    {
                        "!\"": -0.41310275,
                        " to": -2.0537276,
                        ",": -2.2099779
                    }
                ],
                "text_offset": [
                    33,
                    35,
                    36,
                    41,
                    48,
                    57,
                    62,
                    66,
                    72,
                    76
                ]
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 9,
        "completion_tokens": 10,
        "total_tokens": 19
    }
}

This is notably different from the format that I get for streaming responses.
It also highlights that the tests should really mock streaming responses too.

Changing that debug function to just print(response.text) gave me this:
data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":"\n\n","index":0,"logprobs":{"tokens":["\n\n"],"token_logprobs":[-0.3125751],"top_logprobs":[{"\n\n":-0.3125751,"\n":-1.9688251,":":-3.968825}],"text_offset":[33]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":"\"B","index":0,"logprobs":{"tokens":["\"B"],"token_logprobs":[-3.9812593],"top_logprobs":[{"\"B":-3.9812593,"\"":-0.18438435,"\"S":-2.293759}],"text_offset":[35]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":"aking","index":0,"logprobs":{"tokens":["aking"],"token_logprobs":[-0.64748794],"top_logprobs":[{"aking":-0.64748794,"ite":-1.444363,"aked":-1.7568629}],"text_offset":[37]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":" up","index":0,"logprobs":{"tokens":[" up"],"token_logprobs":[-0.21909842],"top_logprobs":[{" up":-0.21909842," with":-2.6878486," happiness":-2.8597233}],"text_offset":[42]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":" pure","index":0,"logprobs":{"tokens":[" pure"],"token_logprobs":[-5.3262787],"top_logprobs":[{" a":-0.8575289," smiles":-1.9825288," happiness":-2.4200287}],"text_offset":[45]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":" ot","index":0,"logprobs":{"tokens":[" ot"],"token_logprobs":[-0.46827942],"top_logprobs":[{" ot":-0.46827942," joy":-1.3276544," delight":-2.8901541}],"text_offset":[50]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":"ter","index":0,"logprobs":{"tokens":["ter"],"token_logprobs":[-0.0015959481],"top_logprobs":[{"ter":-0.0015959481,"terr":-7.126596,"term":-7.845346}],"text_offset":[53]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":" goodness","index":0,"logprobs":{"tokens":[" goodness"],"token_logprobs":[-2.569492],"top_logprobs":[{"-":-1.2726171,"ly":-1.5382422,"lic":-1.8663673}],"text_offset":[56]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":",","index":0,"logprobs":{"tokens":[","],"token_logprobs":[-0.84223104],"top_logprobs":[{",":-0.84223104,"!\"":-1.154731," in":-2.232856}],"text_offset":[65]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":" one","index":0,"logprobs":{"tokens":[" one"],"token_logprobs":[-0.010558619],"top_logprobs":[{" one":-0.010558619," every":-5.432433," bite":-6.3074327}],"text_offset":[66]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":" treat","index":0,"logprobs":{"tokens":[" treat"],"token_logprobs":[-0.31707865],"top_logprobs":[{" treat":-0.31707865," bite":-2.3639536," delicious":-2.6139536}],"text_offset":[70]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":" at","index":0,"logprobs":{"tokens":[" at"],"token_logprobs":[-0.00004501652],"top_logprobs":[{" at":-0.00004501652," a":-10.82817," ot":-11.421921}],"text_offset":[76]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":" a","index":0,"logprobs":{"tokens":[" a"],"token_logprobs":[-0.00006635395],"top_logprobs":[{" a":-0.00006635395," at":-10.390691," ":-10.609441}],"text_offset":[79]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":" time","index":0,"logprobs":{"tokens":[" time"],"token_logprobs":[-0.00006468596],"top_logprobs":[{" time":-0.00006468596," t":-10.562564," tim":-11.51569}],"text_offset":[81]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":"!\"","index":0,"logprobs":{"tokens":["!\""],"token_logprobs":[-0.07164093],"top_logprobs":[{"!\"":-0.07164093,".\"":-2.8060157,"\"":-5.399766}],"text_offset":[86]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80M56thRFHyy1fPDQ79XeMwVGsJ4v","object":"text_completion","created":1695095572,"choices":[{"text":"","index":0,"logprobs":{"tokens":[],"token_logprobs":[],"top_logprobs":[],"text_offset":[]},"finish_reason":"stop"}],"model":"gpt-3.5-turbo-instruct"}

data: [DONE]

I'm going to need tests for the logprobs stuff.
LLM_OPENAI_SHOW_RESPONSES=1 llm -m 3.5-instruct 'say hi, one word' -o logprobs 2

Outputs:
data: {"id":"cmpl-80MdSaou7NnPuff5ZyRMysWBmgSPS","object":"text_completion","created":1695097702,"choices":[{"text":"\n\n","index":0,"logprobs":{"tokens":["\n\n"],"token_logprobs":[-0.61127675],"top_logprobs":[{"\n\n":-0.61127675,"\n":-1.9706517}],"text_offset":[16]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80MdSaou7NnPuff5ZyRMysWBmgSPS","object":"text_completion","created":1695097702,"choices":[{"text":"Hi","index":0,"logprobs":{"tokens":["Hi"],"token_logprobs":[-1.0273004],"top_logprobs":[{"Hi":-1.0273004,"Hello":-0.73042536}],"text_offset":[18]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80MdSaou7NnPuff5ZyRMysWBmgSPS","object":"text_completion","created":1695097702,"choices":[{"text":".","index":0,"logprobs":{"tokens":["."],"token_logprobs":[-1.1168935],"top_logprobs":[{".":-1.1168935,"!":-0.9450184}],"text_offset":[20]},"finish_reason":null}],"model":"gpt-3.5-turbo-instruct"}

data: {"id":"cmpl-80MdSaou7NnPuff5ZyRMysWBmgSPS","object":"text_completion","created":1695097702,"choices":[{"text":"","index":0,"logprobs":{"tokens":[],"token_logprobs":[],"top_logprobs":[],"text_offset":[]},"finish_reason":"stop"}],"model":"gpt-3.5-turbo-instruct"}

data: [DONE]

Hi.

And with --no-stream:
LLM_OPENAI_SHOW_RESPONSES=1 llm -m 3.5-instruct 'say hi, one word' -o logprobs 2 --no-stream

{
  "id": "cmpl-80MeBfKJutM0uMNJkRrebJLeP3bxL",
  "object": "text_completion",
  "created":<PHONE_NUMBER>,
  "model": "gpt-3.5-turbo-instruct",
  "choices": [
    {
      "text": "\n\nHi!",
      "index": 0,
      "logprobs": {
        "tokens": [
          "\n\n",
          "Hi",
          "!"
        ],
        "token_logprobs": [
          -0.61127675,
          -1.0273004,
          -0.9450184
        ],
        "top_logprobs": [
          {
            "\n\n": -0.61127675,
            "\n": -1.9706517
          },
          {
            "Hi": -1.0273004,
            "Hello": -0.73042536
          },
          {
            "!": -0.9450184,
            ".": -1.1168935
          }
        ],
        "text_offset": [
          16,
          18,
          20
        ]
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 3,
    "total_tokens": 8
  }
}

My mocking trick isn't working here (despite working in #287).
https://github.com/openai/openai-python/blob/main/openai/tests/test_api_requestor.py is how OpenAI do it.

My mistake, that trick DOES work, I was using the wrong fixture.

Since I'm already doing this:
            completion = openai.Completion.create(
                model=self.model_name or self.model_id,
                prompt="\n".join(messages),
                stream=False,
                **kwargs,
            )
            response.response_json = completion.to_dict_recursive()
            yield completion.choices[0]["text"]

Which dumps the entire completion.to_dict_recursive() to the DB log for non-streaming responses, I don't need to do anything special to log logprobs for the non-streaming case.
They'll end up in the DB in a slightly different format. I'm OK with that:
{
      "choices": [
          {
              "finish_reason": "stop",
              "index": 0,
              "logprobs": {
                  "text_offset": [16, 18, 20],
                  "token_logprobs": [-0.6, -1.1, -0.9],
                  "tokens": ["\n\n", "Hi", "1"],
                  "top_logprobs": [
                      {"\n": -1.9, "\n\n": -0.6},
                      {"Hello": -0.7, "Hi": -1.1},
                      {"!": -1.1, ".": -0.9},
                  ],
              },
              "text": "\n\nHi.",
          }
      ],
      "created":<PHONE_NUMBER>,
      "id": "cmpl-80MeBfKJutM0uMNJkRrebJLeP3bxL",
      "model": "gpt-3.5-turbo-instruct",
      "object": "text_completion",
      "usage": {"completion_tokens": 3, "prompt_tokens": 5, "total_tokens": 8},
  }

Despite being listed in the documentation, the suffix option does not appear to work - at least not for gpt-3.5-turbo-instruct:
curl https://api.openai.com/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-3.5-turbo-instruct",
    "prompt": "Say this is a test",
    "max_tokens": 7,
    "suffix": "dog",
    "temperature": 0
  }'

{
  "error": {
    "message": "Unrecognized request argument supplied: suffix",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

I managed to submit suffix to an old text-davinci-003 completion model but it didn't seem to have the expected effect.
Dropping suffix support entirely.

I'm going to do the **bold** thing for system prompts, and hope it doesn't turn out to be a bad idea later.

Wouldn't throwing an error be a better alternative? These models aren't designed to use system prompts. It's better to communicate that to end users so that they can adjust their API calls accordingly.

Wouldn't throwing an error be a better alternative? These models aren't designed to use system prompts. It's better to communicate that to end users so that they can adjust their API calls accordingly.

I'm really torn on this.
The reason I'm leaning towards keeping system prompts working is that a really useful application of LLM is to compare the results you get from different models.
It would be frustrating if you tried to compare the results of a prompt with a system prompt and got an error back because one of the dozen models you chose to use didn't support system prompts.
Plus, really when you look at what system prompts actually do in other models, they're basically just injected into the regular prompt with extra markup around them. For Llama 2 that looks like this, for example:
<s>[INST] <<SYS>>
{{ system_prompt }}
<</SYS>>

{{ user_message }} [/INST]

So wrapping the system prompt in bold is actually a pretty honest imitation of how they work everywhere else!

Hmmm... thinking about it, we actually have a bit of a precedent problem here.
llm-claude silently ignores system prompts, which caught me out already - I fed it a system prompt and it didn't work, but I didn't realize because I didn't get an error.
On that basis, an error would actually be a better solution.

Based on that decision in:

#288

I'm going to have this plugin raise an error if you try to send it a system prompt. I'll get rid of that and replace it with the mechanism from #288 once that is implemented.

I don't yet have a pattern for what exception should be raised by a Model.execute() method if something goes wrong.

I'm going to raise NotImplmentedError for this, because the system prompt support is not implemented.


Can't access node attributes

I have an XML feed with attributes on the node elements but can't find a way to access them.
This feature used to work in 2.0.0 beta 1.
Can this be looked into please?
Thanks.

Are you able to provide an example of your feed data? I need some sort of indication on the data that you're trying to map.
Can you also clarify if you are trying to map the node attributes via the control panel, or access them directly via the templates?

Yeah sorry, I'm trying to access the node attributes from within a template.
In the following example,  are the nodes and I need to access their attributes "lat" and "lon".

Active Log: 2016-06-18 07:08

200.57
2016-06-18T08:55:52Z

199.61
2016-06-18T08:55:53Z

198.17
2016-06-18T08:55:54Z

197.69
2016-06-18T08:55:55Z

Sure, so you can use something like:
{% set params = {
    url: 'http://your/url',
    type: 'xml',
    element: 'trkpt',
} %}

{% set feed = craft.feedme.feed(params) %}

{% for node in feed %}
    {{ node['@lat'] }}
{% endfor %}

As a handy hint, you can always use {{ dump(node) }} inside that loop to get an idea of what you have data-wise.

Yay, that works great, but maybe it should be documented?

Actually the documentation was outdated regarding this, so thanks for the prompt to update it! See https://sgroup.com.au/plugins/feedme/feature-tour/using-in-your-templates

Very nice, thank you so much!


Fix port allocation in status tests

Reason for This PR
https://github.com/temporalio/roadrunner-temporal/pull/598
Description of Changes
Status trying to allocate same port for multiple tests
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the MIT license.
PR Checklist
[Author TODO: Meet these criteria.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

[ ] All commits in this PR are signed (git commit -s).
[ ] The reason for this PR is clearly provided (issue no. or explanation).
[ ] The description of changes is clear and encompassing.
[ ] Any required documentation changes (code and docs) are included in this PR.
[ ] Any user-facing changes are mentioned in CHANGELOG.md.
[ ] All added/changed functionality is tested.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 36.89%. Comparing base (c5c2e9a) to head (e9b035a).
Report is 45 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #600      +/-   ##
==========================================
- Coverage   40.06%   36.89%   -3.17%
==========================================
  Files          14       14
  Lines         936     1095     +159
==========================================
+ Hits          375      404      +29
- Misses        518      650     +132
+ Partials       43       41       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Consider using feature flags on the frontend

Sometimes we want to enable certain functionalities in certain environments but disable them in other environments. For example, while we use 2FA on staging and prod, we need a way to avoid that in dev mode. Similarly, we had a discussion on slack recently that we want to test journal export on staging gradually without exposing it to users on prod. Feature flags could be useful in these cases by letting us use the same code in different environments but still helping us control which features we expose to users.
One possible way to implement feature flags would be to have a /metadata endpoint that exposes app metadata (eg: app title etc) along with feature_flags. The React frontend would call this endpoint during initialization and use the values in feature_flags to selectively expose or hide certain feature. For example, the journal export button would be visible only if JOURNAL_EXPORT_ENABLED is set to true in feature_flags and so on.

@bwbaker1 not much for you to review here, but feel free to 😄. This lets us have certain functionality enabled on staging but not on production, which helps with our testing locally -> staging -> production pipeline.


chore: update to react 18

Updates the react library to React 18.
Closes #364

This should be used as the starting point for #364.
Important to note that when resolving conflicts in package-lock.json and other files, you may end up with multiple versions of React installed which will cause some crazy errors. Run npm ls react in the root of the repo to see what versions are being pulled in. If it's inconsistent, then npm update may help.

Taking this out of draft for now. I think that things are working. Editing & Viewing seem to work the same and the tests are passing.
I would love a fairly thorough review to make sure I haven't missed anything.
I also am unsure why tests are not running...

@tuckerzp reopen so I can review


The Footer can be much Better

I will be adding more gaps and margin to the footer so that is look independent from the @ copyright section and will add glassmorphism to the footer so that it will be looking more good

And as you can see that the cursor hide the image of the social  media icon so will be correcting that too.

Assign this Issue to me

The footer has been updated in all the pages same as the homepage.


when use package 'evaluate‘ with 'sacrebleu' calculate metric happend error

original code url:
when i do run a simple case(add some model file) by 'python3 local_evaluation.py'
and when some sample need get bleu metric, encountered such error
 File "xxx/amazon-kdd-cup-2024-starter-kit/local_evaluation.py", line 256, in <module>     main()   File "xxx/amazon-kdd-cup-2024-starter-kit/local_evaluation.py", line 241, in main     per_task_metrics = evaluate_outputs(data_df, outputs)   File "xxx/amazon-kdd-cup-2024-starter-kit/local_evaluation.py", line 99, in evaluate_outputs     metric_score = eval_fn(model_output, ground_truth)   File "xxx/amazon-kdd-cup-2024-starter-kit/local_evaluation.py", line 183, in <lambda>     "jp-bleu": lambda generated_text, reference_text: metrics.calculate_bleu_score(   File "xxx/amazon-kdd-cup-2024-starter-kit/metrics.py", line 254, in calculate_bleu_score     sacrebleu = evaluate.load("sacrebleu")   File "/home/xxx/.local/lib/python3.10/site-packages/evaluate/loading.py", line 751, in load     evaluation_cls = import_main_class(evaluation_module.module_path)   File "/home/xxx/.local/lib/python3.10/site-packages/evaluate/loading.py", line 76, in import_main_class     module = importlib.import_module(module_path)   File "/opt/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module     return _bootstrap._gcd_import(name[level:], package, level)   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked   File "<frozen importlib._bootstrap_external>", line 879, in exec_module   File "<frozen importlib._bootstrap_external>", line 1017, in get_code   File "<frozen importlib._bootstrap_external>", line 947, in source_to_code   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed   File "/home/xxx/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--sacrebleu/009c8b5313309ea5b135d526433d5ee76508ba1554cbe88310a30f85bb57ec88/sacrebleu.py", line 16     } SyntaxError: closing parenthesis '}' does not match opening parenthesis '(' on line 14
It is looklike sacrebleu.py have some problem, maybe my evaluate(0.4.2) or sacrebleu(2.14) version confilict? I dont know
some code in url
def calculate_bleu_score(generated_text: str, reference_text: str, is_japanese: bool = False) -> float:
    """
    Calculates the BLEU score for a generated text compared to a reference truth text. This function supports
    both general text and Japanese-specific evaluation by using the sacrebleu library.

    Parameters:
    - generated_text (str): The generated text to be evaluated.
    - reference_text (str): The reference truth text.
    - is_japanese (bool, optional): Flag to indicate whether the text is in Japanese, requiring special tokenization.

    Returns:
    - float: The BLEU score as a percentage (0 to 1 scale) for the generated text against the reference truth.
    """
    global sacrebleu
    if sacrebleu is None:
        sacrebleu = evaluate.load("sacrebleu")

    # Preprocess input texts
    generated_text = generated_text.lstrip("\n").rstrip("\n").split("\n")[0]
    candidate = [generated_text]
    reference = [[reference_text]]

    # Compute BLEU score with or without Japanese-specific tokenization
    bleu_args = {"predictions": candidate, "references": reference, "lowercase": True}
    if is_japanese:
        bleu_args["tokenize"] = "ja-mecab"
    score = sacrebleu.compute(**bleu_args)["score"] / 100

    return score

 def get_evaluation_methods():
return {
"accuracy": metrics.calculate_per_sample_accuracy,
"hit rate@3": metrics.calculate_hit_rate_3,
"rougel": metrics.calculate_rougel,
"sent-transformer": lambda generated_text, reference_texts: metrics.calculate_cosine_similarity(
generated_text=generated_text,
reference_texts=reference_texts,
model_name="all-MiniLM-L6-v2",
),
"multilingual-sent-transformer": lambda generated_text, reference_texts: metrics.calculate_cosine_similarity(
generated_text=generated_text,
reference_texts=reference_texts,
model_name="paraphrase-multilingual-MiniLM-L12-v2",
),
"micro f1": metrics.calculate_true_positive_false_positives_false_negatives,
"ndcg": metrics.calculate_ndcg,
"bleu": metrics.calculate_bleu_score,
"jp-bleu": lambda generated_text, reference_text: metrics.calculate_bleu_score(
generated_text=generated_text,
reference_text=reference_text,
is_japanese=True,
),
}`

It seems you are using amazon-kdd-cup-2024-starter-kit, which users HuggingFace evaluate, which uses sacrebleu.
If you want to report a bug in this sacrebleu repository, you should show a replicable minimal test case, using sacrebleu API directly (i.e. not via amazon-kdd-cup-2024-starter-kit and evaluate). Otherwise, you should report the bug in one of the above-mentioned frameworks.
That said, I was not able to replicate this bug. Everything seems to work:
!pip install sacrebleu evaluate
import sacrebleu, evaluate
print(sacrebleu.__version__) # 2.4.2
print(evaluate.__version__) # 0.4.2
evaluate_sacrebleu = evaluate.load("sacrebleu")
result = evaluate_sacrebleu.compute(predictions=["John loves Mary."], references=[["John loves HugginFace."]])
print(result["score"]) # 35.35533905932737

I am closing this issue, but feel free to reopen if you identify any bugs in sacrebleu.


Adding isMainThread implementation for Linux

I am going through your articles on KN and I'm compiling the code in this repo on Linux. In the article I noticed this line:

We've also isMainThread, which will require you're doing this on a Mac unless we get a PR that enables other platform. Anyway, the function prints the following.

And it so happens I was just now busy with figuring out how to implement this function on Linux. This works for me:
@SharedImmutable
val isMainThread: Boolean
    get() = getpid().toLong() == syscall(SYS_gettid)

Sample function basicBackground() outputs as expected:
Is main thread true
Is main thread false

I would prepare a PR, but I wonder what's the best way to do this? Maybe it should just be a comment above macOS implementation?
Also I had to replace NSThread.sleepForTimeInterval(2.toDouble()) with sleep(2.toUInt()) in waitForWorker().

I'm going to have to think about main thread for linux. It's not as simple as mac, which simply has one. It probably doesn't make sense in other contexts, but I don't really write native linux apps.

Ok, if you'll need something to be tested and have nowhere to do this, ping me :)
This should be closed then I guess.


Packaging guide

WIP - initial thoughts requested.  I think I will be editing this quite heavily as I was shooting in the dark a bit.

WIP, to be rewritten to include hello-world after it has been updated for v0.3.0


Multiple outputs

First try at allowing filebeat having multiple outputs:

shared inputs; all outputs share the same input config,
named configs; each output has its own main yaml config,
named services; each output has is own systemd unit file,
composite service; to start and stop all named services at once.

Note: only tested on Debian 10.

Thanks @mdklapwijk! This will help a lot moving towards some form of support on #48 too.
It sounds like there are a lot of changes in here in addition to supporting multiple outputs, including deprecating support for Filebeat 6 (probably time to do that anyway), switches to the elastic_stack::repo module for the apt/yum repos, and 2 new module dependencies.
I'd like to add some unit tests to the systemd logic so we can confidently make changes in the future (I need to move them from Travis to GitHub actions so they run here too).

Thanks @mdklapwijk! This will help a lot moving towards some form of support on #48 too.
It sounds like there are a lot of changes in here in addition to supporting multiple outputs, including deprecating support for Filebeat 6 (probably time to do that anyway), switches to the elastic_stack::repo module for the apt/yum repos, and 2 new module dependencies.
I'd like to add some unit tests to the systemd logic so we can confidently make changes in the future (I need to move them from Travis to GitHub actions so they run here too).

Hi Phil,
I was planning on starting a stripped version, dropping FB6, Windows support etc. So I could implement filebeat::output, just like filebeat::input. As stated, this was my first attempt at this and it was done in such a manner it would not interfere with the already existent logic, hence the big foreach-loop over the outputs-hash in config.pp.
If you are interested I would create a new branch for this later this week, as I already (partly) done the filebeat::output, but it broke backward compat. a bit...

Hi Phil,
I was planning on starting a stripped version, dropping FB6, Windows support etc. So I could implement filebeat::output, just like filebeat::input. As stated, this was my first attempt at this and it was done in such a manner it would not interfere with the already existent logic, hence the big foreach-loop over the outputs-hash in config.pp.
If you are interested I would create a new branch for this later this week, as I already (partly) done the filebeat::output, but it broke backward compat. a bit...


Update taggerfactory.py

Fixed Issue #11.

Thanks very much! I have changed the print statement into a logging operation.

Thanks a lot again for sharing this great project!


--ack only support acknowledge true

There is no way to create a stream that doesn't acknowledge.  The default for a stream in true and there is no listed CLI option to set it too false. The use case for this is the need to issue messages that you don't care if they are delivered.  Our case is for Data Analytics for usage tracking of the mobile application.
√ ~ % nats stream add --help
usage: nats stream add [<flags>] [<stream>]

Create a new Stream

Flags:
  -h, --help                    Show context-sensitive help (also try --help-long and --help-man).
      --version                 Show application version.
  -s, --server=NATS_URL         NATS server urls
      --user=NATS_USER          Username or Token
      --password=NATS_PASSWORD  Password
      --creds=NATS_CREDS        User credentials
      --nkey=NATS_NKEY          User NKEY
      --tlscert=NATS_CERT       TLS public certificate
      --tlskey=NATS_KEY         TLS private key
      --tlsca=NATS_CA           TLS certificate authority chain
      --timeout=NATS_TIMEOUT    Time to wait on responses from NATS
      --context=CONTEXT         Configuration context
      --trace                   Trace API interactions
      --config=CONFIG           JSON file to read configuration from
      --validate                Only validates the configuration against the official Schema
      --output=FILE             Save configuration instead of creating
      --subjects=SUBJECTS ...   Subjects that are consumed by the Stream
      --ack                     Acknowledge publishes
      --max-msgs=0              Maximum amount of messages to keep
      --max-bytes=MAX-BYTES     Maximum bytes to keep
      --max-age=""              Maximum age of messages to keep
      --storage=STORAGE         Storage backend to use (file, memory)
      --retention=RETENTION     Defines a retention policy (limits, interest, work)
      --discard=DISCARD         Defines the discard policy (new, old)
      --max-msg-size=MAX-MSG-SIZE
                                Maximum size any 1 message may be
  -j, --json                    Produce JSON output
      --dupe-window=""          Window size for duplicate tracking

Args:
  [<stream>]  Stream name

P.S.  The documentation on on https://github.com/nats-io/jetstream is also out of sync with the CLI tool.

boolean flags like --ack supports for negative --no-ack unfortunately this is not clearly called out in the help due to the design of the cli library

Also just a fiy but JetStream will only send acks to Request() so even if you have a stream with ack on and you dont care for ack's, just use Publish() and you end up with the same outcome

Thanks


Reduce overhead at server side caused by column retrieving from client side

Describe the bug
Current JDBC sends a SELECT query to ClickHouse to retrieve the columns of target table before it executes an INSERT statement.
This SELECT is special designed that it has always false filter. For example
SELECT
  timestamp,
  message
FROM
   store_all
WHERE
  0

If the target table is a distributed table, when ClickHouse receives such SELECT query, it will re-write this statement to query local table and dispatch the re-written SQL to all nodes in the cluster as
SELECT
  timestamp,
  message
FROM
   store
WHERE
  0

However, what we want is the columns of the target table, making the server dispatch this query to all nodes in the cluster is unnecessary.
This problem can be optimized  at the server side. For the server side, obviously, the filter 'WHERE 0' is always FALSE, and no need to dispatch SQL to all nodes in the cluster. But such special predicate judgement before dispatching and re-written might take more time to complete.
Client side can also do such optimization by using DESCRIBE STATEMENT.
Here we're going to get the columns of the target table, in such case, it's the distributed table. DESCRIBE the distributed table is more semantic accurate than executing a SELECT on local table.
So, my suggestion is changing the SELECT statement to DESCRIBE TABLE statement at the client side to reduce extra overhead at the server side.

Thanks @FrankChen021 for the suggestion. It was considered in the beginning but unfortunately DESC does not support modifiers like EXCEPT, not to mention it's tedious to enhance the broken SQL parser to extract all insert columns and cross check with the full list. It would be really nice if ClickHouse supports prepared statement or it can handle the false criteria in a better way,  and it's just not about Java client.
As to the specific case, I'd suggest to use input function instead, so that the type inferring no longer involves the additional query.


[WFCORE-4078]: TimeZone is not correctly taken into account when checking certification expiration

Converting the not After date to the same TimeZone as the current date.
JIRA: https://issues.jboss.org/browse/WFCORE-4078

The failures are related to this PR.

Core - Full Integration Build 7912 outcome was FAILURE using a merge of b5a256bd4cc3b98efc5ad6b3ee24d512f73993f7
Summary: Error while applying patch Build time: 00:00:02

@ehsavoie Just noticed that the ChronoUnit.DAYS.between(current, notAfter) change you had mentioned didn't appear here since the PR was closed.


Install Kubeflow Pipelines using ArgoCD

Background
Currently, the recommended way to install kfp is by directly applying kustomize manifest, checking if
export PIPELINE_VERSION=2.3.0
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"

We will implement install this via argoCD

There's other developments like deployKF but it's too opinionated

i think stick to this method

will not do.


Event stream request is always using GET method

Expected Behavior
When I was making an event stream request, I had chosen to use POST method. But Insomnia made a GET request.
Actual Behavior

Reproduction Steps
No response
Is there an existing issue for this?

[X] I have searched the issue tracker for this problem.

Additional Information
Version: Insomnia 8.2.0
Build date: 10/4/2023
OS: Darwin x64 22.6.0
Electron: 25.8.1
Node: 18.15.0
Node ABI: 116
V8: <IP_ADDRESS>-electron.0
Architecture: x64
Insomnia Version
Insomnia 8.2.0
What operating system are you using?
macOS
Operating System Version
macOS Ventura 13.5.2
Installation method
dmg
Last Known Working Insomnia version
No response

Closing given https://github.com/Kong/insomnia/issues/6682#issuecomment-1759483092

Related: #7182


OpenAi completion models are deprecated

All in the title, text-davinci-003 is deprecated since 4th january, completion models have to be updated to gpt-3.5-turbo

having an issue with this also
tried changing the values in the Custom models field value to [{ "name": "gpt-3.5-turbo 1106", "endpoint":  "https://api.openai.com/v1/chat/completions", "displayName": "gpt-turbo", "model": "gpt-3.5-turbo-1106" }]
is this something that i'd have to fork and modify the source code for?


这个可以配置自己搭建的gitlab  吗

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

可以. 目前进入首先就要配置自己GitLab 的 host 和 token，才能继续使用


Float values limited?

Changing float values seems to be limited to [-0.02f; 0.02f] when changing by holding down the mouse button.
Unity 2021.3.2f1
Rosetta UI 0.0.2 (last version from Package Manager)
P.s. In general, the framework looks very good, thank you very much)

Is it the action of dragging a label?
It is UIToolkit's FloatField and behaves similarly to the Unity Inspector.
You can also change the step width by holding down Shift/Ctrl.
Thank you for your report😀

Example. I increase the float value to the right, but the value does not become more than 0.2f. Looks weird and uncomfortable.
Is this a UIToolkit problem? Everything is fine with int fields
https://user-images.githubusercontent.com/25285476/169289915-68ea6be9-a9b6-468a-868c-32fcd9be1d37.mp4

Thank you for the movie.
I could not reproduce it in my environment.

I don't know the cause of the problem, so I need some information.
If possible, please try the latest version (RosettaUI 0.2.2) or another environment.
If you always have the same problem, please send me a sample project and I will try to reproduce it.

Try a different PC or OS if possible.
I tried it on my M1 Macbook Air and the UIToolkit dragging did not work correctly (but it seems not the same issue).
It worked correctly on Windows.

I made a build (version 0.2.2. Unity version 2021.2.8f), tried to run it on two more PCs, the bug remained. Tried only Windows

I'm sorry, but I can't figure out what's causing this.
If you have any information to reproduce it on my end, please let me know.

Later, when I finish the tasks for my project, I will try to find the causes of this problem and fix it locally. I'll post here if I find a solution.

I have the same issueI have the same issue
I was going through all the examples one by one, and first noticed this at MethodExample, came back to older ones and error is still here
https://user-images.githubusercontent.com/15922601/192898528-cae4756f-d170-4fe3-bad1-9e40636f9440.mp4

I've figured out this issue. The cause is that StringToValue method uses EditorGUI.StringToDouble, which in turn uses double.TryParse without specifying explict culture, so if default system culture formats floats not like 9.5, but for example 9,5 it bugs out

The same issue causes #4


list object has no attribute movedim

when passing the image output onto an upscaler i get the following error

ChatGPT states the following:
The error message 'list' object has no attribute 'movedim' suggests that you are trying to use the movedim method on a list object, which is not supported because lists in Python do not have a movedim method. This error commonly occurs when trying to manipulate the dimensions of a list of tensors.
To resolve this issue:
Check Variable Types: Ensure that the object you are trying to manipulate with the movedim method is indeed a tensor and not a list.
Inspect Data Flow: Trace back the source of the list object and verify its origin. Ensure that the object is properly created and manipulated before attempting to use tensor-specific methods on it.
Debugging: Insert print statements or use a debugger to inspect the type and shape of the object just before the error occurs. This can help identify where the unexpected list object is coming from and why it does not have the expected tensor attributes.
Review Documentation: If you are unsure about the type of object expected at a certain point in your code, refer to the documentation or code comments to clarify the expected data types and formats.
Once you identify the root cause of the issue and correct it, you should be able to resolve the 'list' object has no attribute 'movedim' error. If you need further assistance, please provide more context or code snippets for a more specific analysis.

From what other upscalers are saying, the output from the generator is a tuple, which is not supported by upscalers. Will take a look and see if I can fix

Awesome, thanks!

You have to torch.cat the output images before your return them.
        torch_list = []
        for img in imgs_pil:
            img_tensor = F.to_tensor(img)
            # Ensure the data type is correct
            img_np = img_tensor.permute(1, 2, 0).unsqueeze(0)
            torch_list.append(img_np)
        torch_imgs = torch.cat(torch_list, dim=0)

        return (torch_imgs,)

Thank you @Thater, that worked
Only issue now is, when passed onto vae encode to get some ksampler processing done, it comes up with a different error now, this happens when the vae encoder is directly connected to the generator

Will have to look at vae encoder to see what type of input it takes

So got a bit further in testing this, upscales work, vae encode works (only if a different vae is used.
To me it seems the way the model is implemented along with the vae, this is the cause of the problem here
Once the model or vae from the flash model loader is involved it craps out

Other issue I found was the javascript file in the web folder, with the dynamic input images being populated by this, it does cause issues when joining/disconnecting noodles from the detector node
Would just get rid of that altogether

@GraftingRayman the dynamic inputs are a bit wonky, I could remove them and allow either a single image or an image list or image batch as input instead.

just remove the dynamic input, works fine as it is without, can use a joiner to join lots of images or load a batch, can load a list even with a list to batch node in between.
This helps troubleshooting as can see what dimensions the outputs are, with the dynamic inputs it tries to connect to the dynamic and creates a loop

Dynamic inputs removed


Add integration tests to test helm+flux

See the README for more details.  This fixes #919, and adds tests for flux+helm.  Besides the tests copied from test-flux (test that the sync tag gets updated, test that automation works) it adds tests to ensure that charts in the git repo get deployed, that changes made to charts in the git repo result in a helm release upgrade, and that helm changes made outside of git get reverted.

since I rely on the ssh-known-hosts configmap being mounted

Ahh yes I see, that does make it a bit tricky. Maybe @stefanprodan will be kind enough to comment here on how it could be adapted to serve both purposes (I know he is a fan of mounting known_hosts from a configmap, from #1113).
Aside from converging on the Helm chart, it would be really nice if we can get this to run in CI -- that would help avoid the situation that prompted it (the prior tests rotting). Can it be done with the minikube --vm-driver=none do you reckon? We've used this with some success elsewhere: https://github.com/weaveworks/launcher/blob/master/.circleci/config.yml#L87

since I rely on the ssh-known-hosts configmap being mounted

Ahh yes I see, that does make it a bit tricky. Maybe @stefanprodan will be kind enough to comment here on how it could be adapted to serve both purposes (I know he is a fan of mounting known_hosts from a configmap, from #1113).

The helm chart now has a parameter for supplying a host key, which gets mounted into the fluxd container. Is that enough to rework this PR around the chart herein? (I will attempt so, eventually, but feel free to beat me to it :-)

The -profile bug in minikube appears to have been fixed in 0.28.1 -- I've had some success reworking this branch a bit to use that version.

I'm finally coming back to this patch.  Thanks for pointing out the fix in 0.28.1.
I will look into working with the upstream charts soon.  For now my focus has been on getting this working with TravisCI, as I agree with you that should be done to prevent rot.  Yes, I'm almost certain it can be done with minikube --vm-driver=none.  Main change required is that since we don't have a host node to use as our git repo host any longer, I'm deploying a pod to be the git server instead.  I've got it working that way locally.  In travis I've got it to the point where kubernetes comes up and we can interact with it, but the tests are still timing out while waiting to see the flux-sync tag get updated.  I expect I should have that fixed and an updated PR issued within the week.

@ncabatoff Great! For testing against self-hosted repos, I mashed together https://github.com/squaremo/git-serv at one point -- I can adapt it to what's needed here, if that would be helpful.

I've been continuing my experiments in my flux-tester repo: I feel a bit more free to do whatever since it's more isolated from your repo and I still don't really know what I'm doing git-wise.
Sadly I fear circleci is not going to be viable for running k8s integration tests, at least with minikube.  I really wanted it to work, because the config format seems nicer and better documented and the system is much more responsive and reliable, based on the little I've used it so far.
However:

CircleCI 1.0 will not be supported as of August 31
It's unclear what support they plan to provide for building in a VM environment instead of a docker container, especially at the free tier (they hint it may start costing money at some point)
They've responded to requests for Ubuntu 16 with "use a docker container in CircleCI 2.0"; it sounds like they'd prefer to drop the VM option
Minikube with vm-driver none seems to depend on systemd and won't run in Ubuntu 14

An alternative might be https://github.com/kubernetes-sigs/kubeadm-dind-cluster, but it doesn't look anywhere near as stable or well supported as Minikube.
My experience today with travis, incidentally (it ran builds every time I tried a tweak to the circleci config.yml) was pretty positive today: out of 9 builds, 7 passed.  Unfortunately I think it's a bit hit or miss, but it may still be our best option for now.

Via @hiddeco: looks like it may still be possible to run minikube --vm-driver=none in CircleCI 2.0, but it needs a particular container image (not the Go image we use for building the binaries). I wonder if we can use jobs or workflows or whatever to do it. https://github.com/gavinzhou/ci-minikube

Rebased and modestly worked-on branch at git checkout pr/1101. I realised that working integration tests are a lot of value for #1382, and don't have to work in CircleCI straight away.

We have the Flux dev meeting coming up next week. One of the action item from last time was to take a look at this effort and see if the remaining work could be broken up into smaller pieces.
In broad strokes: what's still missing and where could people help?

We now have an end-to-end smoke test, which does not quite exercise as much of flux as here, but is likely to be the base on which more end-to-end tests are added.


Import from API scripts with data cleansing

Pivotal story: https://www.pivotaltracker.com/story/show/94589786
import_services_from_api script rewritten with dmutils.apiclient
import_suppliers_from_api script with data cleaning
Reads suppliers from one API endpoint and writes to another one.
Recursively replaces all fields named 'email' and 'phoneNumber'
with placeholder values in:

main supplier fields
fields containing nested dictionaries
fields containing lists of dictionaries

An option to create users when importing suppliers
Calling import_suppliers_from_api with --user=PASS options will
create user accounts for each supplier, using PASS as password
and<EMAIL_ADDRESS>as email.

The two new importers look good to me. It would be good to rationalise the naming of the import scripts. The more recent scripts are quite obvious but the earlier ones are not very explicit.


Optimized transitions for cranelift

This bug is in the context of use of Cranelift to generate wasm sandboxed libraries for use in Firefox - i.e. this is a non web embedding. Full details available here https://bugzilla.mozilla.org/show_bug.cgi?id=1562797
We are using lucet as our wasm compiler which uses Cranelift under the covers. For performance reasons, we need to optimize transitions between native code and wasm sandboxed code beyond what lucet provides out of the box. Key to this is generating efficient trampoline functions that minimize overhead.
To this end we have rewritten trampolines that make a few assumptions based on " properties derivable from the WASM spec about register and stack use of a WASM compiler". These assumptions reduces overhead when using a sandboxed version of libGraphite by several 100's of percent.
Since this is something indirectly implied from the WASM spec, I am hoping that Cranelift can guarantee these and wanted to confirm if these are OK.
There are 3 assumptions - one about stacks, and two about register use

Assembly functions generated by Cranelift WASM never underflows the stack

This is guaranteed from the WASM spec due to the fact that WASM functions are proper stack machines
This would allow us to use the same stack for the application and wasm module
However, our reliance on this depends on how bug-free we think Cranelift's conversion from the WASM stack machine

Assembly functions generated by Cranelift never reads an uninitialized register

This is an outcome of all function calls to WASM and within WASM (including indirect jumps) being typesafe.
This means there should never be code generated that reads from a scratch register prior to writing to it.
This also means there should never be code generate for a function that takes 2 parameters, that attempts to read a value from the register that would hold a third parameter (if it had existed)
This allows us to avoid clearing scratch and unused parameter registers during WASM function invocation.

Assembly functions generated by Cranelift always restore callee save registers

Like point 2, this is an outcome of all functions being typesafe combined with the fact that WASM is a stack machine, which cannot manipulate the stack in a way that affects register spills.
This means that there should no path through the function that would result in a callee save register not being restored
This allows us to avoid explicitly restoring registers on the way out

It looks like this issue's purpose was mainly to establish answers to the above questions; it looks like there is also a pending question of whether the invariants should be documented. However, it doesn't strike me that they are particularly noteworthy beyond what is already documented: the question is basically whether Cranelift follows the calling convention (restores callee-saved registers), and whether the dataflow of its output is correct (never reads an uninitialized register). These fall out of "correct compiler that follows the spec" and aren't any special additional guarantee. So I'll go ahead and close, but please feel free to ask for other clarifications as needed!

Sounds good. I had initially raised this question as we are relying on a performance optimization that made the above assumptions. If it is of interest, we have subsequently shown with formal semantics, that any spec conforming Wasm compiler must automatically meet the above restrictions, as well as the workloads and performance benefits we get. Details available here


Photos stuck in processing since August.

http://openstreetcam.org/details/512250
http://openstreetcam.org/details/512248
http://openstreetcam.org/details/512244

Yeah, I'm up to 27 stuck in processing. All 3 of my uploads today got stuck. What is really annoying me is that I can't delete tracks that are stuck in processing. There are several that are failed uploads and I would like to get rid of them.

Some tracks are stuck in "processing" for more than a year.
Would it be possible to re-start processing of all unprocessed tracks that were uploaded more than 3 months ago?

It appears to me that this issue and #207 are the same, with #207 being a bit newer, with more activity. Shall we close this one so that people can follow events under #207?


UAVCAN update

@davids5, this update fixes https://github.com/UAVCAN/uavcan/issues/8. Should it be rebased to nuttx_bringup?

Rebased and merged.


AppVeyor: Use existing Go 1.53 instead of downloading + installing

@maxtaco

Should we go ahead and update to 1.6? We've already done it on Linux. (And OSX?)

I have no issues with 1.6

If we wanted to do anything complicated with specific versions of Go, we could consider testing many of them at once :-D This probably matters more on the Linux side of things, though, where kind of care about Arch users building the client on their own machines.

Updating to Go 1.6 would be very nice.

On the 6th or 7th try, it passed without modification. Merging.


合并4.8.1版本

Description

Addressed Issue

Additional Details

Checklist

[ ] I have read and understand the contributing guidelines
[ ] This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

合并4.8.1版本


Added TrackPointUtils for customizable movable average

Customizable Moving Average Algorithm
This class implements a utility for calculating the moving average of speeds from track points. It is based on a user selection of 5,10,15 data points and a 5-second interval.
Note: new dependencies/libraries
Please refrain from introducing new libraries without consulting the team.

Closes #16


Adapt IR for stricter pattern binding rules coming in Scala 3.2

Ref https://github.com/lampepfl/dotty/pull/14294#issuecomment-1017281801
This is a blocker for the Scala 3 PR enabling stricter pattern binding rules.

Could we improve the commit message? IMHO the title should say what is changed (something like "use @unchecked;l for destructing assingments...") the body should say why.

@griggt Could you amend the commit message as requested by @gzm0 ? We're approaching the 2 months since the last release, so we will soon publish a new release. It would be good to get this PR in.


Not possible to change different numbers on same line.

Yo,
I really enjoy using boole so thanks for creating something I didn't know I needed :). Today I've noticed something interesting though.
When  or  a number (for example mb-10 ml-4 mr-2) It automatically jumps to the mb-10 word and <inc/dec> it. If I then move my cursor to ml-4 it actually jumps back to mb-10 and <inc/dec> that one again. This pattern repeats for other numbers in the same line.
Any thoughts on this? Is this behavior intended, or is this something which has not been noticed before?
Keep up the good work. Cheers.

That's a bug. I'll look into it. Thanks for the kind words and the report.

That's cool, thanks!
Something else I've noticed. When for example changing bg-blue-500 when my cursor is on the 5 it actually increases the 500 to 501. Same happens when cursor is on the first 500. Is that intended?
Imho it would make sense to only inc/dec the number targeted by the cursor or selection?

Just to inform but this behavior is still present even after the merge.
It happens whenever I am inside a className tag. For example.
<div className="my-2 mb-10 bg-blue-500">
When incrementing it will auto jump to my-2 and increment it to my-3, but there i no way to target mb-10 or bg-blue-500 since it auto jumps to my-2

Thanks for reporting this. I think I finally got the bug, but I will leave this issue open for you to close when you have time to pull the latest version and test it yourself.

Looks perfect! Thanks. Not sure if I can close though, the button is gone now.


delete postProcess.ts as it is not needed

What is the purpose of this PR?
To remove postProcess.ts, as the logic contained in this class is carried out by transcriptToCuts.ts. The double-up occurred to a communication lapse between agile teams, so this PR is just here to amend that.
What was changed in this PR?
postProcess.ts is deleted and the Clip interface removed from sharedTypes.ts

Some impressive stuff here Liam. Keep it up.