[Request - Help Wanted] Add Mixcloud support (play while downloading)

It would be nice to have Mixcloud support on the bot. Right now if someone tries to play a mixcloud mix the bot stops working and needs a restart.

Don't think its that useful. Most of the songs you can find on youtube.

Many DJs have weekly podcasts on their Mixcloud accounts and the tracks there are mixed together. Playlists have huge delay between songs. Also, Mixcloud includes live performances of many artists that you can't find on YouTube or SoundCloud. And many bots already incorporate Mixcloud support so I don't see a reason not to implement it.

Oh well... this is the first time i hear about Mixcloud

Update: Mixcloud seems to be actually working, though the whole track needs to be downloaded. It can't play otherwise. After waiting for the whole track to download you have to disconnect the bot and summon it and play. Investigating.

Update 2: It is normal behavior to not start playing until the whole track is downloaded, but it seems mixcloud tracks download 4 megabytes and then stop. Restarting the bot and playing the same track downloads another 4 megabytes and so on...

Update 3: This weird behavior is caused by Mixcloud throttling speeds after approximately 5 megabytes of continous download. This is intended so that a stream downloads while you're listening.
From the current implementation the bot waits for the track to finish downloading before playing.
If that could be changed so that the bot streamed the audio while it was being downloaded would solve the issue.
Tried with different download options and downloaders and the best-case scenario was to speed-download the first time and then restart the bot and play from there (it sees the file and thinks it's already downloaded and starts playing it).
Marking as [Help Wanted]

Closing for inactivity


Not able to set fixed height.

 RichHTMLEditor(html: $data.text, textAttributes: textAttributes)
                            .editorScrollable(true)
                            .onEditorLoaded {
                                // Perform action when editor is loaded
                            }
                            .onCaretPositionChange { newPosition in
                                // Perform action when caret moves
                            }
                            .onJavaScriptFunctionFail { error, function in
                                // Perform action when an editor JavaScript function has failed
                            }
                            .introspectEditor { richEditorView in
                            }
                            .focused($isFocused, equals: true)
                            .padding(.vertical, dynamic(5))
                            .padding(.horizontal, dynamic(12))
                            .overlay(
                                    RoundedRectangle(cornerRadius: dynamic(cornerRadius))
                                          .stroke(data.isFocused ? theme.primary : theme.dividerColor, lineWidth: 1)
                            )
                            .frame(height: dynamic(height))

https://github.com/user-attachments/assets/a3e25592-d937-46b0-9ad7-79936013d15f

@valentinperignon @JorisBodin Can you please share any fix for this issue?

Hi, version 1.2.0 should fix your issue. Please feel free to reopen this issue in case something is still not working as expected.

Thank you for the quick fix. It worked perfectly!

I have been using this library for my macOS app, the function
func editorScrollable(_ scrollable: Bool) -> some View {         environment(\.editorScrollable, scrollable)     }
is not accessible for AppKit.
If is there any possible work-around so that I can enable scroll functionality in editor.
@PhilippeWeidmann @chitraarasu @chitraarasu-kirshi

Hi @sandeep-singsys,
Since this is a different topic, could you create a separate issue?
We will look into this.


Unit test failures: Async#retry

https://dev.azure.com/monacotools/Monaco/_build/results?buildId=189155&view=logs&j=a5d99aad-4929-5289-bf4a-26b78032353d&t=c61a688d-0a79-5b1f-223b-9376911e515d

There is a bunch of other test failures (see https://github.com/microsoft/vscode/issues/165278 and https://github.com/microsoft/vscode/issues/162102) that only occur on macOS and only in web tests, though both in Chromium and Webkit. The symptoms are always the same: a seemingly random test times out that works fine in electron tests.
I can only speculate that this is an issue with setTimeout not firing so here in this case I can attempt to fix it with runWithFakedTimers but thats not really bringing us closer to understanding why this actually happens...


Clarify Handling of globalAssetId and specificAssetId (Constraint AASd-116)

Is your feature request related to a problem? Please describe.
Handling of globalAssetId and specificAssetId is not completely clear in combination with. Background Part 1:
Background Part 1 Metamodel:
Both attributes are optional but with  the following note:
The asset has a globally unique identifier, plus – if needed – additional domain-specific (proprietary) identifiers. However, to support the corner case of very first phase of life cycle where a stabilized/constant global asset identifier does not already exist, the corresponding attribute "globalAssetId" is optional.
and the following constraints:
Constraint AASd-116: "globalAssetId" (case-insensitive) is a reserved key. If used as value for SpecificAssetId/name, SpecificAssetId/value shall be identical to AssetInformation/globalAssetId.
Constraint AASd-131: The globalAssetId or at least one specificAssetId shall be defined for AssetInformation.
Describe the solution you'd like
Decision  Proposal from TF AAS API from 2024-03-27 and approved by TF Metamodel from 2024-03-27

Add to Part 1 Constraint AASd-116 that the SpecificAssetId with "name" = "globalAssetId" shall only be used for the Discovery operations but not for the creation of Asset Administration Shells or SubmodelElements of type Entity.

Describe alternatives you've considered
Use Case: Create a new AAS (in bold the one that would be the expected behavior for the recommended solution:
A) Create a new AASDescriptor or AAS:

Case1: The caller sends the globalAssetId as part of the specificAssetIds but not as part of the shell

We could set the globalAssetId of the shell to the same value

We could throw an error that they are not identical or missing

Throw an error , reason:

Violation of AASD-116: stating that the name globalAssetId must not be used for SpecificAssetIds, or

Violation of AASd-131: stating that there must be at least one valid entry for globalAssetId or specificAssetIds if no other other SpecificAssetId is defined

Case 2: The caller sends the globalAssetId as part of the shell but not as part of the specificAssetId

We could add the globalAssetId to the specifcAssetId

and also return it as part of specificAssetId in shell

remove it from specificAssetiIds

Not add it as part of the specificAssetIds  but support search for it on shell level but not on specificAssetId

Case3: The caller sends the globalAssetId as part of the specificAssetId and as part of the shell but they are different

We could throw an error that they are not identical

Throw an error , reason:

Violation of AASD-116: stating that the name globalAssetId must not be used for SpecificAssetIds

Case4: The caller does not send any info for the globalAssetId

We could throw a warning that there is no global asset ID defined

Throw an error, reason:

Violation of AASd-131: stating that there must be at least one valid entry for globalAssetId or specificAssetIds if no other other SpecificAssetId is defined

Remove attribute globalAssetID and force a (Key,Val) tuple in specific Asset ID that has the keyword "globalAssetID".

2024-04-12 Workstream AAS: approved

In combination with https://github.com/admin-shell-io/aas-specs/issues/298 the proposal is to state the following:
Constraint AASd-116: : "globalAssetId" (case-insensitive) is a reserved key for SpecificAssetId/name with the semantics as defined in https://admin-shell.io/aas/3/0/AssetInformation/globalAssetId.
Note: AASd-116 is important to enable a generic search across global and specific asset IDs (e.g. in IDTA-01002-3-0 discovery operations like GetAllAssetLinksById). In the future the constraint might become more strict in stating that the name “globalAssetId” shall not be used as SpecificAssetId/name.

[...] "globalAssetId" (case-insensitive) [...]

Please note that the case-sensitivity can not be reliably implemented across different cultures (good example is Turkish because of the i, I, İ and ı). You can specify a regular expression, specifying exactly which characters are meant, though.

Workstream AAS Specs
Change to
Constraint AASd-116: : "globalAssetId" (case-insensitive) is a reserved key for SpecificAssetId/name with the semantics as defined in https://admin-shell.io/aas/3/0/AssetInformation/globalAssetId.
Note: AASd-116 is important to enable a generic search across global and specific asset IDs (e.g. in IDTA-01002-3-0 discovery operations like GetAllAssetLinksById). In the future the constraint might become more strict in stating that the name “globalAssetId” shall not be used as SpecificAssetId/name.

solved in IDTA-01001-3-0-1: https://industrialdigitaltwin.org/wp-content/uploads/2024/06/IDTA-01001-3-0-1_SpecificationAssetAdministrationShell_Part1_Metamodel.pdf


BREAKING CHANGE: deprecate 'suiteparams' infavor of 'params'

fix #288
Deprecates -s, --suite-params in favor of -p, --params, will be removed before GA as heartbeat still uses the --suite-params flag.

:grey_exclamation: Build Aborted

Either there was a build timeout or someone aborted the build.'}

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #331 opened

Start Time: 2021-06-30T18:37:37.218+0000

Duration: 28 min 16 sec

Commit: 83aa540ce6bd6f092c2e6634e3bfa326c22754f2

Test stats :test_tube:

Test
Results

Failed
0

Passed
113

Skipped
0

Total
113

Trends :test_tube:

Log output
Expand to view the last 100 lines of log output

[2021-06-30T18:53:44.488Z]  > git checkout -f 408d725dcbc79fee27b89f54b9d4278826a23ad0 # timeout=15
[2021-06-30T18:53:48.016Z] Commit message: "Merge commit '05fad12c65efe7f4dda0a1488ab64f555b844c0d' into HEAD"
[2021-06-30T18:53:48.016Z] First time build. Skipping changelog.
[2021-06-30T18:53:48.016Z] Cleaning workspace
[2021-06-30T18:53:48.095Z]  > git --version # timeout=10
[2021-06-30T18:53:48.103Z]  > git --version # 'git version 2.17.1'
[2021-06-30T18:53:48.114Z] fatal: bad object f9f89c7688f920e3ef655b7e4e87fec857123e29
[2021-06-30T18:53:48.019Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-30T18:53:48.035Z] Resetting working tree
[2021-06-30T18:53:48.036Z]  > git reset --hard # timeout=10
[2021-06-30T18:53:48.058Z]  > git clean -fdx # timeout=10
[2021-06-30T18:53:49.937Z] Running in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331/src/github.com/elastic/synthetics
[2021-06-30T18:54:03.952Z] + set +x
[2021-06-30T18:54:03.952Z] => Downloading nvm from git to '/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331/.nvm'
[2021-06-30T18:54:03.952Z]
=> Initialized empty Git repository in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331/.nvm/.git/
[2021-06-30T18:54:04.890Z] From https://github.com/nvm-sh/nvm
[2021-06-30T18:54:04.890Z]  * [new tag]         v0.37.1    -> v0.37.1
[2021-06-30T18:54:04.890Z] => Compressing and cleaning up git repository
[2021-06-30T18:54:04.890Z]
[2021-06-30T18:54:04.890Z] => Profile not found. Tried ~/.bashrc, ~/.bash_profile, ~/.zshrc, and ~/.profile.
[2021-06-30T18:54:04.890Z] => Create one of them and run this script again
[2021-06-30T18:54:04.890Z]    OR
[2021-06-30T18:54:04.890Z] => Append the following lines to the correct file yourself:
[2021-06-30T18:54:04.890Z]
[2021-06-30T18:54:04.890Z] export NVM_DIR="$HOME/.nvm"
[2021-06-30T18:54:04.890Z] [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[2021-06-30T18:54:04.890Z]
[2021-06-30T18:54:06.269Z] => Close and reopen your terminal to start using nvm or run the following to use it now:
[2021-06-30T18:54:06.269Z]
[2021-06-30T18:54:06.269Z] export NVM_DIR="$HOME/.nvm"
[2021-06-30T18:54:06.269Z] [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[2021-06-30T18:54:06.528Z] Found '/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331/src/github.com/elastic/synthetics/.nvmrc' with version <12.18.4>
[2021-06-30T18:54:06.788Z] v12.18.4 is already installed.
[2021-06-30T18:54:07.047Z] Now using node v12.18.4 (npm v6.14.6)
[2021-06-30T18:54:07.047Z] /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331
[2021-06-30T18:54:07.168Z] Running in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331/src/github.com/elastic/synthetics
[2021-06-30T18:54:07.491Z] + chmod -R ugo+rw examples
[2021-06-30T18:54:07.521Z] Running in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331/src/github.com/elastic/synthetics/__tests__/e2e
[2021-06-30T18:54:07.844Z] + npm run test
[2021-06-30T18:54:07.844Z]
[2021-06-30T18:54:07.844Z] > e2e-synthetics@ test /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331/src/github.com/elastic/synthetics/__tests__/e2e
[2021-06-30T18:54:07.844Z] > npm i && npm run setup &&  bash scripts/test.sh
[2021-06-30T18:54:07.844Z]
[2021-06-30T18:54:11.138Z] added 3 packages from 4 contributors and audited 178 packages in 2.52s
[2021-06-30T18:54:13.673Z]
[2021-06-30T18:54:13.673Z] 68 packages are looking for funding
[2021-06-30T18:54:13.673Z]   run `npm fund` for details
[2021-06-30T18:54:13.673Z]
[2021-06-30T18:54:13.673Z] found 0 vulnerabilities
[2021-06-30T18:54:13.673Z]
[2021-06-30T18:54:13.673Z]
[2021-06-30T18:54:13.673Z] > e2e-synthetics@ setup /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331/src/github.com/elastic/synthetics/__tests__/e2e
[2021-06-30T18:54:13.673Z] > bash scripts/setup.sh
[2021-06-30T18:54:13.673Z]
[2021-06-30T18:54:13.673Z]
[2021-06-30T18:54:13.673Z] Temporary folder
[2021-06-30T18:54:13.673Z] Temporary files will be stored in: ./tmp
[2021-06-30T18:54:13.673Z]
[2021-06-30T18:54:13.673Z] Starting elasticsearch , kibana and synthetics docker
[2021-06-30T18:54:13.673Z]
[2021-06-30T18:54:13.673Z]
[2021-06-30T18:54:13.673Z] Waiting for synthetics docker to start...
[2021-06-30T18:54:13.673Z] Error: No such object: synthetics
[2021-06-30T18:54:23.652Z] Error: No such object: synthetics
[2021-06-30T18:54:33.647Z] Error: No such object: synthetics
[2021-06-30T18:54:45.873Z] Error: No such object: synthetics
[2021-06-30T18:54:54.129Z] Error: No such object: synthetics
[2021-06-30T18:55:04.122Z] Error: No such object: synthetics
[2021-06-30T18:55:14.133Z] ✅ Setup completed successfully. Running e2e tests...
[2021-06-30T18:55:16.171Z] Waiting for Elastic Search  to start
[2021-06-30T18:56:12.663Z] Waiting for Synthetics to send data to ES for test monitor
[2021-06-30T19:05:40.447Z] Cancelling nested steps due to timeout
[2021-06-30T19:05:40.452Z] Sending interrupt signal to process
[2021-06-30T19:05:50.271Z] scripts/test.sh: line 26:  5273 Terminated              SYNTHETICS_JUNIT_FILE='junit.xml' npx @elastic/synthetics uptime.journey.ts --reporter junit
[2021-06-30T19:05:50.271Z] npm ERR! code ELIFECYCLE
[2021-06-30T19:05:50.271Z] npm ERR! errno 143
[2021-06-30T19:05:50.271Z] npm ERR! e2e-synthetics@ test: `npm i && npm run setup &&  bash scripts/test.sh`
[2021-06-30T19:05:50.271Z] npm ERR! Exit status 143
[2021-06-30T19:05:50.271Z] npm ERR!
[2021-06-30T19:05:50.271Z] npm ERR! Failed at the e2e-synthetics@ test script.
[2021-06-30T19:05:50.271Z] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
[2021-06-30T19:05:50.271Z]
[2021-06-30T19:05:50.271Z] npm ERR! A complete log of this run can be found in:
[2021-06-30T19:05:50.271Z] npm ERR!     /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331/.npm/_logs/2021-06-30T19_05_45_909Z-debug.log
[2021-06-30T19:05:50.277Z] script returned exit code 143
[2021-06-30T19:05:51.628Z] Post stage
[2021-06-30T19:05:51.662Z] Archiving artifacts
[2021-06-30T19:05:51.815Z] Recording test results
[2021-06-30T19:05:51.930Z] None of the test reports contained any result
[2021-06-30T19:05:51.939Z] [Checks API] No suitable checks publisher found.
[2021-06-30T19:05:52.203Z] Stage "Release" skipped due to earlier failure(s)
[2021-06-30T19:05:52.243Z] Stage "Release" skipped due to earlier failure(s)
[2021-06-30T19:05:52.324Z] Stage "Release" skipped due to earlier failure(s)
[2021-06-30T19:05:52.678Z] Running on Jenkins in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-331
[2021-06-30T19:05:52.764Z] [INFO] getVaultSecret: Getting secrets
[2021-06-30T19:05:52.811Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-30T19:05:53.661Z] + chmod 755 generate-build-data.sh
[2021-06-30T19:05:53.662Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-331/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-331/runs/1 ABORTED 1696172
[2021-06-30T19:05:53.662Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-331/runs/1/steps/?limit=10000 -o steps-info.json
[2021-06-30T19:05:53.662Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-331/runs/1/tests/?status=FAILED -o tests-errors.json


integration tests cleanup

Motivation and Context

Housekeeping
Description

This organizes some things, removes some unused things, and adds a missing feature to some integration tests
Testing

I ran all tests that these changes touched
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

can you make sure that running cargo test in that directory still works with the deleted lib.rs files?

I'm confused as to why this is failing when running either of these commands locally succeeds:

cargo test (in the s3control integration test directory)
./gradlew :aws:sdk:cargoTest (from smithy-rs root dir)

I'm going to move on to other, more important tasks and return to this after reinvent


Only validate signatures and IPs when request path matches hooks endpoint.

Not sure if that's okay, but it would allow to use the HookServer with other routes like an about page or something instead of having provide a separate app for that.

This looks good. Thanks!


Pulling in new OpenAPI /api into our new new docs site

Preview is available here:http://developers-pr2.previews.kube001.services.stellar-ops.com/docs


completeOrder of Order class requires session

The function that finalises the order after a successful transaction relies on the session to find details about the order to complete.
If the order either creates a new user, or someone is already logged in, this will not update all details on the user correctly when an external payment method is used (i.e. Paypal).
So this needs to be refactored so it can be triggered from any source and session info is not required.

I've marked this as a bug, as I don't believe Paypal (or similar functioning gateways) will work correctly until this is resolved.

I think the issue is 2 fold:

What if a person cancels on the payment gateway
The webhook from the online payment gateway does not contain the session

Problem 1 has an easy fix : Add a cancelled page below the checkout.
Problem 2 is a bit more tricky...
The problem resides in the session not being available in the webhook, which is to be expected since the payment gateway (for example the paypal server) does not have the order session.
Solution: Check the payment not on the webhook, but on the returnURL.

On the notifyURL the transactionID is set on the order but not completed and not marked as paid.
register a route to a custom method in the payment gateway class (on_start method in the controller.php) .
The returnURL goes to the newly registered route.
Customer returns after completion to the returnURL -> order session is available again.
Because the transactionID was saved on the notifyURL we can ask the payment gateway, what the status is of the transaction with that ID.

if paid -> completeOrder and mark as paid -> redirect to checkout/complete

if open (not paid, not cancelled) -> completeOrder and mark is unpaid -> redirect to checkout/complete

if cancelled -> redirect to checkout/cancelled

I'm not familiar with the paypal API so I'm not sure this will apply to paypal...  This does work on a different online payment method I am currently working on (and hopefully be releasing soon).

My working example may take some time ... I forked the paypal payment gateway to my repositories and made the changes I described above. However, the call to paypal to check the payment status has not been implemented. Click here to see the forked repo
Note: this is theoretical, I have no idea if it will work. On the other hand, going from session to something else is a huge change.

Thanks @Jozzeh, I think you make a really good point about a cancelled transactions, we're probably missing a screen to handle that sort of scenario.
With handling the check of payment on the returnURL, I see that as a potential problem as there's no guarantee that the customer won't just close their paypal window (or whatever external gateway), immediately after they've made their payment. In that case you've got a transaction that has happened but an order that hasn't been marked as paid.
On the other hand the webhook back to the site is considered to "always" occur, something that is going to ensure that the order is marked as paid. This passes through the order id, so I think this method is going to going to be fine.
What this issue specifically refers to is that as part of the order completion process it's still looking at some details from the session, to make updates to the related user.  I've got a feeling the Paypal add-on for community store does actually work correctly, it's just that you can't use it to create new user accounts or update user groups.

@Mesuva You are completely correct about the webhook. It should always occur.
I do think the webhook (with the current implementation), should only be used for marking as paid and not to complete an order.
Paypal will indeed work , but a lot of things will not fire without the session. For example:

Order Emails won't be send
Cart won't be cleared
Unsetting the shipping type
Create new user account/update user groups (as you've mentoined)

By not having the above, an online shopper will be left confused about what happened with his/her order and payment.
A more robust solution is necessairy. So your first proposition when starting this issue (refactoring/rethinking) seems the best option.
A fairly easy fix could be to save the customer details in the database and linking it with a certain order when the payment starts. (Sort of like drafts but for customers instead op pages)

@Jozzeh I'm seeing what you mean here. I think we could use the webhook to trigger off the final process of the order completing, which will send emails and adjust user groups, but you're right in that the session is needed to clear the cart, unset the shipping type and log in the new user account.
I think we could start off by adjusting the order completion step to use as little as possible from the session, then we could split off the session related stuff to a different function/process.
Something to do with a good chuck of spare time and a strong coffee!

After looking into this , I think I have found a way. Just like you said, splitting off the session related stuff to a different function/process.
I've noticed the orderComplete function now uses no session except for :

Discount Code clear
community_store.smID
Cart clear

I've written a new payment gateway (still in test, Mollie payment gateway) but I've solved the issue by registering an extra route in the controller.php of the package. So I have 2 routes registered in the controller.php ... one for the webhook and one for the returnUrl.
public function registerRoutes(){
    Route::register('/checkout/mollieresponse', '\Concrete\Package\CommunityStoreMollie\Src\CommunityStore\Payment\Methods\Mollie\MolliePaymentMethod::validateCompletion');
    Route::register('/checkout/ordercompletion/{oID}', '\Concrete\Package\CommunityStoreMollie\Src\CommunityStore\Payment\Methods\Mollie\MolliePaymentMethod::customerValidation', 'customervalidate' ,array('oID' => '\d+'));
}

public static function customerValidation($oID){
   //Sends customers to this function when the transaction is completed or cancelled
   //this function checks if the payment was successfull or not using the $oID to get the order
   //depending on the payment status , the user will be redirected to the appropriate page
}

This returnUrl route goes to a function in the payment method and actually has the order ID as a parameter in the Url.
In this function, I first check if the payment was successfull/paid (getting the transaction ID from the order).
If successfull -> clear the session stuff and redirect the user to /checkout/complete .
If cancelled or not paid -> redirect to /cart
I will try to upload the new payment gateway as soon as possible.
If paypal has the ability to check a payment status outside of the webhook, this might solve the issue.  Even though we are not sure a user will actually return to the webshop after the payment, the user will get an e-mail with the order details (because the orderComplete function is done in the webhook) .

@Jozzeh I've reworked the order process here: https://github.com/concrete5-community-store/community_store/commit/4fcecbca545f26aa9b10d0fedecf6375739d7294
It needs to be tested further, but I believe I've split out the order process so that it doesn't require session data at all to complete. It also means that we should be able to add a 'resend invoice' function to the dashboard.
Will test this further, but you're welcome to have a look at my changes so far

I've had a quick look at the changes and it looks excellent.
Will test it as soon as possible.
The resend invoice might be a good enhancement to the dashboard, in case an invoice mail is lost.

I've finished this up and merged it, including a resend invoice function.
I've tested this with Stripe, Paypal and Pin Payment gateways, as well as the Invoice payment, for normal and membership products, logged in and logged out. All seemed correct to me.
The recent change also means that if someone puts through an invoice order for a membership product, they wont get their login immediately, but when the order is marked as paid by an admin it will complete the rest of the process (new user account, membership groups)


InputLayout dose not clear error when EditText is blank.

I am using the code from the sample to an email and password field , when I type something it pops up error but when I clear the filed and even when I focus to the next object, InputLayout keep showing the error unless I type an email.

and also carbon_label disappears when field is not focused anymore.

Label automatically disappears when it's set to floating style. Use 'app:carbon_labelStyle="persistent"' to get a persistent label.
Empty email is not valid, so the behavior is correct. InputLayout clears error state only if the text matches the set pattern.

app:carbon_labelStyle="persistent" put the label to the top always. What if I want to make my label style floating by default and only when edittext is not empty it will change to persistent style ?

I've added new style called ifNotEmpty which does exactly what you suggested. I'm not sure if the name is the best, but it should do the job :)


Browser Compatibility

We should test
Linux
EndeavourOS (LeSnake)

[x] Firefox fresh profile
[x] Firefox Hardened profile
[ ] Chromium – Crahes after a  few seconds with error code SIGIL
[ ] Brave  – Crashes when spamming bullets – Code SIGIL

Windows

[ ] Firefox
[ ] Brave
[ ] Chrome
[ ] Edge

MacOS
(does anyone of us has a mac, a hackintosh or lives with someone with a mac?)

[ ] Safari
[ ] Brave
[ ] Firefox
[ ] Chrome

closing this because normally wasm build work on all modern browsers. check caniuse.com


Update Readme.md file

Description:

Suggested Labels:

Suggested Assignees:

Affected Product Version:
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:

Fixed with PR #201


[Suggestion] Transparency and customization of the coin address derivation process via network parameters

I would like to formally propose an intention towards increasing the transparency of the cryptocurrency address derivation process (from either BIP32 keys or private/public keys). There are two separate issues:

there's no clearly defined protocol (with some self-evident verification) behind how new coins are supposed to be integrated (e.g. via Pull Requests), such that anyone could (learn to) do it,
there's no way to use all of the coin parameters manually, as the fields aren't exposed and editable, and so everyone has to rely on the official support.

This is a follow up to some ideas mentioned in and after my https://github.com/iancoleman/bip39/issues/378#issuecomment-561595912. My idea is that this tool is slowly gaining the responsibility over teaching the basic guidelines of how the wallets work, including it being the only "proper open-source and offline" tool to derive, and also generate (using a custom entropy with the option of not trusting any single RNG - e.g. my idea is to XOR multiple providers yourself) all of the keys required to operate and backup wallets, using the HD best practices (which also enables sharing a single seed to be used for multiple coins - though this idea doesn't seem to be promoted, because after all, importing an important seed to a third party wallet app isn't intrinsically safe - maybe unless you use a hardened child key directly, which doesn't seem to be advised either). This project is even referenced in The Bitcoin Book: Mastering Bitcoin as a de-facto standard solution.
As far as I know, there are two separate concerns when considering the usage of a new coin (I'm not an expert, so I'm looking forward to a confirmation), and the user should be able to completely verify all of them (e.g. via official coin documentation) without trusting any third party (alternatively, they can do a dry run to test both public and private keys of some low-value wallet using a wallet app):

they have to support the same public key cryptography form (algorithm, length...) (I think this is a straightforward binary choice, either they extend Bitcoin, or they require a custom implementation),
the network (base58/bip32 (public, private)/pubKeyHash/scriptHash/WIF...?) parameters need to be correct (the end users only validate this by checking the first character of the public Address, so there may be a danger if this is a mistake).

The second requirement is straightforward, but this also shows the ever-increasing need to consider the first concern. For example, there's a SmartCash coin based on Keccak (SHA-3). They have made their own fork of this project at github.com/xdustinface/bip39-smart, which is also hosted at hdwallet.smartcash.cc or hdwallet.dustinface.me. After a Discord discussion of the option to back-integrate their project, I've found out that it's not a simple matter of changing the network parameters, as they have their own JavaScript to derive this address. Thus, I was thinking that this coin could serve as a test to validate both of my requirements.
Considering that it could be dangerous to back-integrate random third party scripts in the future though, I also have one more question: is there a way to sandbox the script such that, for example, it cannot access anything but a single Public Key? I'm sure some people would appreciate the option to maximize the security, even if only for explicitly manual purposes (I'm sure the well known algorithms will need to be integrated anyway).
Thanks for your time, and please correct me if I'm missing some easier solution :)

I'll also provide some random references for people interested in this topic, as I had to spend a great deal of time before having found out that it's not as intuitive as it seems, and there's no starting point:

Wallet parameters are usually hard-coded in their source code, e.g. https://bitcointalk.org/index.php?topic=225690.0 and https://bitcointalk.org/index.php?topic=3345808.0
Address prefixes can vary under different circumstances: https://en.bitcoin.it/wiki/List_of_address_prefixes
Bitcoin parameters under chainparams.cpp (at v.0.14), though often it's under base58.h: https://github.com/bitcoin/bitcoin/blob/0.14/src/chainparams.cpp#L129-L131
Validity check of a wallet address could be non-standard if the Base58Check format weren't followed: https://en.bitcoin.it/wiki/Base58Check_encoding
Addresses can have different prefixes if they are scripts (P2SH) vs. if they are hashes of wallet public keys (P2PKH), or public keys directly (P2PK) bitcoinwiki.org/wiki/Pay-to-Script_Hash || Why is P2PKH used instead of the simpler P2PK?
HD (BIP32) process explained: https://www.youtube.com/watch?v=OVvue2dXkJo

it's not a simple matter of changing the network parameters, as they have their own JavaScript to derive this address.

See ripple, ethereum, stellar and others in libs directory for others that have this issue and have found ways to solve it.
Generally I'm happy to include third party libraries, since the npm and github reporting of affected versions is good.

it could be dangerous to back-integrate random third party scripts in the future

Yes I have always felt this, so I'm very open to ideas around how to improve this.

is there a way to sandbox the script such that, for example, it cannot access anything but a single Public Key?

Not that I know of.

I'm sure some people would appreciate the option to maximize the security

From my perspective this comes from the surrounding environment (OS, browser, offline, memory wipe on reboot etc). The tool can never really be fully trusted since it's extremely difficult to audit the entire codebase and the entire toolchain.
As a general comment I agree it would be good to streamline the process of adding new networks.
It may also be possible to allow users to add their own custom parameters in the UI but I feel that's a last ditch thing. My feeling is if a coin community can't put in the effort to submit a PR they're probably not a community that should be using this tool.

Personally I'd really prefer the option to use the custom parameters, mainly because I feel that nothing should be a black box when it comes to cryptocurrencies. I know it shouldn't really be your responsibility, but it's exactly the consequence of the cryptocurrency community looking at such transparency as a last ditch thing :P Even if the coin is already supported, I'd still prefer to have the option of entering its parameters myself. There's also the motivation to (subconsciously) educate users, because there's (generally in software) currently a dangerous trend of hiding any real "issues" along with vital facts, and forcing inexperienced users to use inferior workflows as a default, so they don't even know what they're doing. I've hoped in evolution of cryptocurrencies in the opposite way, such that it'd be easier for example to use cold wallets to sign, or even create transactions, and use open-source tools to verify and transmit the tx without ever exposing the seed to them... Nowadays you're lucky if a client works offline at all.

Unrelated: (UX of Windows 10 vs. 98 is a nice example of this trend, as even "My Computer" was removed from a desktop, and instead of using disk paths directly, casual users are left to hope that shortcuts like Downloads/Pictures will take care of their needs. Now they even went to the extreme of creating a "Storage Sense" feature that removes all old files automatically, so who knows what'll happen next...) As a reference, there's also the cognitive theory of internalization vs. externalization.

Either way, exposing the network parameters is only the first step, because it would make using private/public keys way more obvious. For example, if they have some hardened derivation key, users should be able to conceptually understand which keys are able to be used to derive addresses (it should be easy if they know all of the coin parameters), and which of them could be used to manipulate with their wallet (e.g. as long as official wallets use a WIF format, it would be nice to provide transformation steps at least for well known coins, though the perfect solution would be if the wallet apps supported private keys directly). I could also imagine various private (blockchain) projects, or experimental projects that are currently under development. Last but not least, there are projects just like this SmartCash, which don't really care about being back-integrated, because they don't share (my) transparency ideals, and their viewpoint is this simple: "Most users will probably use the web version." I believe there'll always be many projects that prioritize use-cases of masses, and will rely on their own implementations, which nobody will ever complain about. That's why I believe it should be up to users if they want an absolute security, and not up to the good will and free time of developers of the coin :)
When it comes to integrating algorithms like Keccak, if it isn't really reasonable to provide any sandbox, then any simple deterministic PR guideline could be enough (such that even people like me, who haven't worked with Keccak at all, will find it easy to merge a third party project). As long as the algorithm is a public parameter though, I have an idea that the JS could be at least included dynamically (e.g. after refreshing a page) and unloaded after switching to a different algorithm. This could even enable a workflow of copying a public key, loading a script, and translating the key to an address. Though such a paranoid approach may be an overkill, after all, this script is supposed to be used offline :) But I think that exposing algorithms via single <select> tag wouldn't have any downsides. I hope this can be easy to implement in a near future :)

Some other things I want to add (just making a note for myself for later) regarding quality control

metadata for the coin website which allows automatic periodic checking if the site still exists, to remove old unused dead coins
metadata for the coin web wallet, or at least a compatible wallet that allows testing with a third party


Web links are rendered as regular text.

Describe the bug
Web links within the docstring are rendered as regular text rather than a web link.
To Reproduce
Steps to reproduce the behavior:
def docstest(int_val: int) -> int:
    """Function to showcase docstring problems.

    This docstring demnonstrates some issues using ``mkdocstrings`` with respect to hyperlinks. They will get rendered as regular text.

    Parameters
    ----------
    int_val
        Integer value. Please see https://en.wikipedia.org/wiki/Integer.

    Returns
    -------
    int
        Returns an integer value.

    """
    pass

Expected behavior
https://en.wikipedia.org/wiki/Integer should get rendered as a web link.
Screenshots

System (please complete the following information):

griffe version: 0.22.2
Python version: 3.10

Try adding pymdownx.magiclink to your Markdown extensions in mkdocs.yml:
markdown_extensions:
- pymdownx.magiclink

Please ask for reopening if the above suggestion does not solve your issue.


Fix compile issue with Segments.hpp

Thanks. fixed by https://github.com/BYVoid/OpenCC/commit/88d58640ab57faeed89546ec8089a0fefdef6b41


Allow to not attach a Rack to a Device

Hi,
I would like to add the CPE Devices (Customer side) but I must select a Rack. Is it possible disable this ?
Thanks in advance.
Thomas V.

Duplicate of #198


Failure screenshot hidden away in Execution test body

Hello,
I'm using Allure along with WDIO.
One thing I need help with - for failing tests, the screenshot is hidden in the Execution test body - attaching a screenshot to explain what i mean.
How do i get the screenshot to appear at the top?

Hello! Current behavior is defined in our code and cannot be changed on your side.
However we would be happy to accept a PR, that improves functionality about this. The code which is responsible for that feature is around this line
If you have any questions, please let us know!


[WIP] Refactor types module to subcrate

Start of the work on #941 to give an idea to the maintainers if they like it.
Marking this as a draft for that reason.
This does not compile, it still needs some work on the features side and
transfers of those features from said crate.
There are other issues that has cropped up since the refactoring, namely
of non-exhaustive matches, but those are most likely due to the lack of
feature management.

I may be wrong but it seems that git-mv has not been used.
Consequently, all history for files in rusqlite-types/src is lost !

Hm my bad, I tried fixing that but I seem to have ammended the wrong commit. I'll close this.


[CWM-192] Add etcd to support IAM with minio gateway.

Add etcd single node in docker-compose to support IAM.
Resolves https://github.com/CloudWebManage/cwm-worker-cluster/issues/192.

Signed-off-by: Azeem Sajid<EMAIL_ADDRESS>

made some doc changes, made etcd server optional


Deleting features from imported JSON

I can not delete features from an uploaded JSON file

Please upload a JSON with the issue.

Unable to reproduce; closing until this comes up again.


Allow changing columnWidth and columnSpacing on Table

At the moment it doesn't seem possible to change the columnWidth or columnSpacing at a later time. If the data is changed then setData blows up because it checks columnWidth in the initial options that are passed.
The use case for this is that when my Table widget changes size I meant want to add/remove a column to make it responsive.
I can try and make a PR for this if want?

This would be really nice, as well as accepting percentage values instead of absolute numbers (i.e. 3 columns of 60% / 20% / 20% width of the table full width)


Some issues while working with partitioned table

We should modify readme to not use it with partitioned table :) atleast for now/unless we address these issues :) .
When I inserted some data on my partitioned table and after that just quickly ran pgosc on that partitioned table in order to check out how does it work.. it actually , created another normal table -added column there but didn't copy the data
-- Here are the tests:
-- my partitioned table with some data:
CREATE TABLE shiwangini.events2 (
  device_id bigint,
  event_id bigserial,
  event_time timestamptz default now(),
  data jsonb not null,
  PRIMARY KEY (device_id, event_id)

)PARTITION BY Hash (device_id);

create table shiwangini.events2001 partition of shiwangini.events2 FOR VALUES WITH (modulus 4, remainder 1);
create table shiwangini.events2002 partition of shiwangini.events2 FOR VALUES WITH (modulus 4, remainder 2);
create table shiwangini.events2003 partition of shiwangini.events2 FOR VALUES WITH (modulus 4, remainder 3);
create table shiwangini.events2004 partition of shiwangini.events2 FOR VALUES WITH (modulus 4, remainder 0);

INSERT INTO shiwangini.events2 (device_id, data)
SELECT s % 100, ('{"measurement":'||random()||'}')::jsonb FROM generate_series(1,100000) s;

-- I can see the data now:
table shiwangini.events2 ;

Now, I ran below pgosc command to add a column.. the statement executed successfully on console:
pg-online-schema-change perform --alter-statement 'alter table events2 add column "name" varchar ;' --schema "shiwangini" --dbname "dev" --host "xx.xx.xx.xx"  --username "dev"  --pull-batch-count 1000 --delta-count 20   --wait-time-for-lock 5
{"name":"pg-online-schema-change","hostname":"dev-host","pid":1714106,"level":40,"time":"2022-03-19T17:28:55.211+00:00","v":0,"msg":"DEPRECATED: -w is deprecated.  However, when I manually check schema of events2 table - I see name column has been added.(but, I don't see any data there)
-- shiwangini.events2 definition

-- Drop table

-- DROP TABLE shiwangini.events2;

CREATE TABLE shiwangini.events2 (
	device_id int8 NOT NULL,
	event_id int8 NOT NULL DEFAULT nextval('shiwangini.pgosc_st_events2_f6ffc9_event_id_seq'::regclass),
	event_time timestamptz NULL DEFAULT now(),
	"data" jsonb NOT NULL,
	"name" varchar NULL,
	CONSTRAINT pgosc_st_events2_f6ffc9_pkey PRIMARY KEY (device_id, event_id)
)
WITH (
	autovacuum_enabled=false
);

However , I see another (probably old table with the name : pgosc_op_table_events2) - which has old data and old partitions and exactly old schema.  If I specify --drop in the end , the old table with all data get dropped :D

Yeah, currently this does not work with partitioned table where child table inherits from parent tables. Since, pg-osc uses shadow tables for its procedure, adding a column to a shadow table, then attaching that partition to a child table against this new shadow table won't work because schema would look different.  We'd need shadow-copy/migrate each child table in order to support schema changes for partitioned tables.

cc @jfrost if you have any thoughts/ideas here.

Closing this issue now - updated readme to indicate partitioned tables are not supported for now.

yeah, same! I will get back if/when there is a good alternative


Expose RetryClient externally

Greetings,
This is a bit far-reaching PR. We're using https://github.com/grafana/terraform-provider-grafana which uses this library for Grafana OnCall API.

terraform-provider-grafana has an option called insecure_skip_verify which allows to use the provider when hosting Grafana on self-signed (/self-hosted) certificates.
Unfortunately, I've recently discovered that the insecure_skip_verify option that is defined, used in parseTLSConfig method which is used only once  for configuring main Grafana API client, which in turns mean that we cannot use the provider to connect to Grafana OnCall API with self-signed certificates.

As terraform-provider-grafana uses this library, the easy way to change that is to expose RetryableHTTP client field publicly, so users of this library (terraform-provider-grafana) can use the same parseTLSConfig and pass the appropriate TLS settings to underlying Go HTTP Client.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


adding a basic npm linting workflow for ui dir

/cc @nerdalert
Relates to: #317

Ready for Review, the current state of UI is broken. Let me know if it is in scope for me to fix it.


pip coverage rate cannot be collected

Describe the bug
i code some self test case about pip. and i want to cout coverage of pip, but  maybe it is some wrong
To Reproduce

cd /usr/lib/python3.9/site-packages/pip
coverage run __main__.py install numpy

Expected behavior
cout coverage
Additional context
[root@localhost pip]# coverage run __main__.py install numpy
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
Looking in indexes: http://mirrors.tools.douban.com/pypi/simple
Collecting numpy
  Downloading http://mirrors.tools.douban.com/pypi/packages/f8/ea/ff38168d6565a8549f819699cac4d89bbc38fc5b27fb94f8e92bcd713348/numpy-1.23.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     |████████████████████████████████| 17.1 MB 6.2 MB/s
Installing collected packages: numpy
Successfully installed numpy-1.23.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
/usr/local/lib64/python3.9/site-packages/coverage/control.py:794: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

If I understand correctly, you are trying to measure the code coverage of the pip test suite itself?  How are its tests usually run? Do they not use coverage already in their configurations?

yes, you are right, but if i input pip install numpy, and i only want to the command's coverage. how do i use?
as we know, if i code test.py, and i measure the file of coverage, i can input coverage run test.py, so i input coverage run __main__.py install numpy, why does it output No data was collected.", slug="no-data-collected

It will depend on how you have configured coverage measurement in your .coveragerc file.  Try coverage run -m pip install numpy perhaps?

it still is not ok, maybe it is some wrong with pip

I still want to understand what you are trying to do: are you making a change to pip and want to run its test suite? Have you read this?  https://github.com/pypa/pip/blob/main/docs/html/development/getting-started.rst#running-tests

yes, i read it, but it is a bug with the test suite of pip, https://github.com/pypa/pip/issues/11415
and the pip measure  coverage with pytest-cov in its test suite,  and i found it is some wrong with pytest-cov, https://github.com/pytest-dev/pytest-cov/issues/552
so i try to measure the coverage with coverage, But it doesn't seem to be going well.

You've linked to pypa/pip#11415, which clearly shows coverage data being collected for pip, so I'm going to close this issue.  It sounds like you need to write some tests for the pip test suite.


Warning in custom sysimage

When backing PlutoPlotly into a custom sysimage that contains Pluto and PlutoPlotly, I get the following warning upon starting Pluto
┌ Warning: You loaded this package outside of Pluto, this is not the intended behavior and you should use either PlotlyBase or PlotlyJS directly
└ @ PlutoPlotly ~/.julia/packages/PlutoPlotly/8WjBh/src/PlutoPlotly.jl:12

Does is_inside_pluto() not work as intended in this case?
Steps to reproduce on julia 1.7.3
using Pkg
Pkg.activate(temp = true)
Pkg.develop("PrecompilePlutoCourse")
cd(joinpath(Pkg.devdir(), "PrecompilePlutoCourse", "Demo"))
Pkg.activate(".")
using Demo
Demo.create_sysimage()
Demo.start()

I think this happens because when you are precompiling during creation of the sysimage the code is ran outside of Pluto and you get the warning.
Did you try to see if the precompiled image actually fails to create plots in Pluto?

The precompiled image works fine and creates plots in Pluto. It's just the warning that looks puzzling :)

I'll probably have to reword the warning to mention this can also happen during precompilaton for the sysimage

I realized the printing was actually removed in https://github.com/JuliaPluto/PlutoPlotly.jl/commit/b792f2725676218f18f458dbfd772549df5854bc before v0.3.0.
Where you using the latest PlutoPlotly version?


kontrol usability issues

When calling foundry-view-kcfg and there are several versions of the same proof, a Python error gets propagated to the top.
It would be a lot nicer if we printed a helpful error message, and listed the available IDs (maybe even with the last time it was run, or some other useful data that can help the user understand which is which).
Traceback (most recent call last):
  File "/nix/store/p08ari7cc5qn9gy3s1vwfqr9pzi7scpc-python3.10-kevm-pyk-1.0.291/bin/.kontrol-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/p08ari7cc5qn9gy3s1vwfqr9pzi7scpc-python3.10-kevm-pyk-1.0.291/lib/python3.10/site-packages/kontrol/__main__.py", line 71, in main
    execute(**vars(args))
  File "/nix/store/p08ari7cc5qn9gy3s1vwfqr9pzi7scpc-python3.10-kevm-pyk-1.0.291/lib/python3.10/site-packages/kontrol/__main__.py", line 265, in exec_foundry_view_kcfg
    foundry = Foundry(foundry_root)
  File "/nix/store/p08ari7cc5qn9gy3s1vwfqr9pzi7scpc-python3.10-kevm-pyk-1.0.291/lib/python3.10/site-packages/kontrol/foundry.py", line 74, in __init__
    with (foundry_root / 'foundry.toml').open('rb') as f:
  File "/nix/store/syz2y6j53y5hpzbs7l0965zwxshi8iyl-python3-3.10.10/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: 'foundry.toml'

When you get a non-deterministic branch and click it in the TUI, you get a crash:
╭────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
│ /nix/store/3q2jq2933abmcn832icdj8fd00pzcxc4-python3.10-pyk-0.1.434/lib/python3.10/site-packages/pyk/k │
│ cfg/tui.py:400 in on_graph_chunk_selected                                                             │
│                                                                                │
│   397 │   │   elif message.chunk_id.startswith('ndbranch_'):                                          │
│   398 │   │   │   self._selected_chunk = None                                                         │
│   399 │   │   │   node_source, node_target, *_ = message.chunk_id[8:].split('_')                      │
│ ❱ 400 │   │   │   source_id = int(node_source)                                                        │
│   401 │   │   │   target_id = int(node_target)                                                        │
│   402 │   │   │   ndbranch = single(self._kcfg.ndbranches(source_id=source_id, target_id=targe        │
│   403 │   │   │   self.query_one('#node-view', NodeView).update(ndbranch)                             │
│                                                                                │
│ ╭─────────────────────────────────── locals ───────────────────────────────────╮                      │
│ │           _ = ['9']                                                          │                      │
│ │     message = Selected()                                                     │                      │
│ │ node_source = ''                                                             │                      │
│ │ node_target = '8'                                                            │                      │
│ │        self = APRProofViewer(title='APRProofViewer', classes={'-dark-mode'}) │                      │
│ ╰──────────────────────────────────────────────────────────────────────────────╯                      │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: invalid literal for int() with base 10: ''

Reproduction:

clone https://github.com/smart-transaction/test-contract
checkout branch hjort
 kevm foundry-kompile --verbose && kevm foundry-prove --test PinataTest.testA0 --smt-timeout 300 --smt-retry-limit 4 --verbose
open with kevm foundry-view-kcfg
locate a non-deterministic branch (I see one after node 8).
click it

In the TUI, it should be displayed which test and ID you're looking at. I sometimes want to compare KCFGs in different windows, or leave the computer and come back, so it's nice to have the context of what you're looking at without having to kill the TUI and open it back up.

It should be possible to reload the TUI without quitting it and reopening: just hit Shift+R or similar to add new nodes.

A Kontrol config
It is very error prone to keep in mind what command you just kompiled with, which modules are imported, etc.
Instead of requiring the user to always kompile with certain flags, we should offer a configuration file (similar to how foundry does, which lets you set things like number of retries, etc). If you need to try using different flags here and there that can be managed with profiles.
This is also how Foundry does it, so it would be a recognizable pattern for devs.
(As a small bonus, seeing commited kontrol.toml files in Solidity git repos would work as good marketing, the same way you see Dockerfiles all over and that immediately tells you "oh this team is using Docker".)
So instead of for example
kontrolx foundry-kompile --require "aux-lemmas.k" --module-import PinataTest:AUX-LEMMAS --rekompile --with-llvm-library --verbose --regen

and then
kontrolx foundry-prove --max-iterations 50 --verbose --no-break-on-calls --test PinataTest.testA0 --max-depth 1000

we could have a file kontrol.toml:
[profile.a0]
test = 'PinataTest.testA0'
max-depth = 1000

kompile-require = [ 'aux-lemmas.k' ]
kompile-module-import = [ 'PinataTest:AUX-LEMMAS' ]
kompile-llvm-library = true

prove-max-iterations = 50
prove-break-on-calls = false

This would likely reduce errors due to forgetting a certain flags, and improve reproducibility since every engineer should be able to get the same result by just doing
kontrolx foundry-prove --profile a0

The best order of priority is probably

command-line flags
config file
defaults

TODO: make these separate issues

@hjorthjort Is there anything that you need the Kontrol team to do on this Issue at this time, or should we wait for the separate issues referenced above before we move to act on them? We have this listed as "Under Investigation" until that is complete, but please let me and @palinatolmach know if you need more support and we are happy to provide it.

I believe a kontrol config may be necessary pretty soon for customer satisfaction, and their ability to reliably reproduce results. But that is a hunch from my side, not something any user has expressed the need for.

I've broken this into separate issues.


recent_topics: Show loading indicator before fetching initial messages.

Show the same loading indicator we show for messages in recent topics,
before any message is fetched. After first message is available
for processing, hide the loading indicator.

What does the position: absolute CSS added to the common .required::after class do? I'd like to understand why that won't mess up the settings pages that also use that component.

Moved that change to its own commit. It avoids a visual change to the column widths. This css won't apply to settings.
before:

after:

Is the !initial_loading_indicator_hidden && topic_data_changed check correct? I worry that we'll show a loading indicator perpetually if you're in an organization with no messages in topics.

Added a comment to address it and removed the topic_data_changed condition.

@amanagr could you please post a screenshot with the loading indicator? Thanks!

I showed @alya the UX and that part is approved; so we just need to clean up the implementation as described above and this should be good to merge.

@timabbott updated as per your review.

Excellent, merged, thanks @amanagr!


Moving a torrent won't prevent PicoTorrent from downloading the torrents again in their old location

So I just discovered that if you move a torrent (In the UWP version at least) and then you restart the program, PicoTorrent will attempt to download the torrent from scratch again in the old location.
Super annoying.

Hi! Do you move the file on disk or are you using the context menu in PicoTorrent?

Context menu in PicoTorrent
Right click a torrent -> move

Odd, that's a bug indeed. I'll definitely have a fix for it in the next version. Thanks for you patience!

Should be fixed in #486


fix: Unsupported platform tag macosx_14_0_arm64 (#14700)

fix: Unsupported platform tag macosx_14_0_arm64 (#14700)

Thanks! Looks like we need the test updated as well.
Would you mind updating this line to major version 15?
https://github.com/pypi/warehouse/blob/main/tests/unit/forklift/test_legacy.py#L3143


s/getTestPathPattern/getTestSelectionConfig

Codecov Report

Merging #4003 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master    #4003   +/-   ##
=======================================
  Coverage   60.26%   60.26%
=======================================
  Files         196      196
  Lines        6765     6765
  Branches        6        6
=======================================
  Hits         4077     4077
  Misses       2685     2685
  Partials        3        3

Impacted Files
Coverage Δ

packages/jest-cli/src/lib/update_argv.js
95% <ø> (ø)
:arrow_up:

...ages/jest-cli/src/lib/get_test_selection_config.js
52.94% <ø> (ø)

packages/jest-cli/src/run_jest.js
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a1ffbb...336734a. Read the comment docs.


Incredibly slow response time

Hello.
I am still new to llama-cpp and I was wondering if it was normal that it takes an incredibly long time to respond to my prompt.
Fyi, I am assuming it runs on my CPU, here are my specs:

I have 16.0Gb of RAM
I am using an AMD Ryzen 7 1700X Eight-Core Processor rated at 3.40Ghz
Just in case, my GPU is a NVIDIA GeForce RTX 2070 SUPER.

Everything else seems to work fine, the model could be load correctly (Or at least, it seems to be).
I did a first test using the code showcased in the README.md
from llama_cpp import Llama
llm = Llama(model_path="models/7B/...")
output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
print(output)

which returned me this:

The output is what I expected (Even though Uranus, Neptune and Pluto were missing), but when I see the total time, it is extremely long (1124707.08ms, 18 minutes).
I did this second code in order to try a bit to see what could be causing the insanely long response time but I don't know what's going on.
from llama_cpp import Llama
import time
print("Model loading")
llm = Llama(model_path="./model/ggml-model-q4_0_new.bin")

while True:
    prompt = input("Prompt> ")
    start_time = time.time()

    prompt = f"Q: {prompt} A: "
    print("Your prompt:", prompt, "Start time:", start_time)

    output = llm(prompt, max_tokens=1, stop=["Q:", "\n"], echo=True)
    print("Output:", output)
    print("End time:", time.time())
    print("--- Prompt reply duration: %s seconds ---" % (time.time() - start_time))

I may have done things wrong since I am still new to all of this, but do any of you have any idea on how I could speed up the process? I searched for solutions through google, github and different forums, but nothing seems to work.
PS: For those interested in the CLI output when it loads the model:
llama_model_load: loading model from './model/ggml-model-q4_0_new.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: type    = 1
llama_model_load: n_parts = 1
llama_model_load: type    = 1
llama_model_load: ggml map size = 4017.70 MB
llama_model_load: ggml ctx size =  81.25 KB
llama_model_load: mem required  = 5809.78 MB (+ 2052.00 MB per state)
llama_model_load: loading tensors from './model/ggml-model-q4_0_new.bin'
llama_model_load: model size =  4017.27 MB / num tensors = 291
llama_init_from_file: kv self size  =  512.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

I apologize in advance if my english doesn't make sense sometimes, it is not my native language.
Thanks in advance for the help, regards. 👋

Hi @MartinPJB , it looks like the package was built with the correct optimizations, could you pass verbose=True when instantiating the Llama class, this should give you per-token timing information.
Also, if possible, can you try building the regular llama.cpp project and trying out those examples just to confirm that this issue is localized to the python package.

Hi @MartinPJB , it looks like the package was built with the correct optimizations, could you pass verbose=True when instantiating the Llama class, this should give you per-token timing information.
Also, if possible, can you try building the regular llama.cpp project and trying out those examples just to confirm that this issue is localized to the python package.

Hi @abetlen, I did pass verbose=True when instantiating the Llama class:
llm = Llama(model_path="./model/ggml-model-q4_0_new.bin", verbose=True)

I assume it is what you meant. However, it is not printing any details in the console after the submission of my prompt.

Since you said you built the package with the correct optimizations, maybe it is coming from my computer itself, however, I do not have any idea why it is so slow.
By the way, I am not familiar with building the regular llama.cpp project. Maybe you could provide some help or other issues related to that.

If you follow these instructions you should be able to test it out https://github.com/ggerganov/llama.cpp#usage

No worries. I am gonna do what's needed and share you the results in a bit, if that is good for you.

For what it's worth, I am also getting a lot slower generations vs. natively interacting with llama.cpp in the terminal

@abetlen  I did some testing on my M1 Mac, and here are my results (llama.cpp is faster)
Stats for calling /v1/chat/completions
llama_print_timings:        load time =  3102.15 ms
llama_print_timings:      sample time =   189.52 ms /    57 runs   (    3.32 ms per run)
llama_print_timings: prompt eval time = 10289.07 ms /    61 tokens (  168.67 ms per token)
llama_print_timings:        eval time =  8149.10 ms /    56 runs   (  145.52 ms per run)
llama_print_timings:       total time = 18632.98 ms

Llama.cpp
llama_print_timings:        load time = 19000.44 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings: prompt eval time = 18459.57 ms /   512 tokens (   36.05 ms per token)
llama_print_timings:        eval time =  3828.14 ms /     1 runs   ( 3828.14 ms per run)
llama_print_timings:       total time = 22828.60 ms

Here are the model loads
llama-cpp-python
llama_model_load: n_vocab = 32001
llama_model_load: n_ctx   = 2048
llama_model_load: n_embd  = 5120
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 13824
llama_model_load: n_parts = 2
llama_model_load: type    = 2
llama_model_load: ggml map size = 7759.84 MB
llama_model_load: ggml ctx size = 101.25 KB
llama_model_load: mem required  = 9807.93 MB (+ 1608.00 MB per state)

llama.cpp
llama_model_load: n_vocab = 32001
llama_model_load: n_ctx   = 2048
llama_model_load: n_embd  = 5120
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 13824
llama_model_load: n_parts = 2
llama_model_load: type    = 2
llama_model_load: ggml map size = 7759.84 MB
llama_model_load: ggml ctx size = 101.25 KB
llama_model_load: mem required  = 9807.93 MB (+ 1608.00 MB per state)

Thoughts? Anything you

What's the difference between prompt_eval time and eval time? seems like llama.cpp is higher on eval time for some reason 🤷

Ran a couple more runs for llama.cpp with --mlock (using 13B model) and got some better (?) results
llama_print_timings:        load time = 15164.63 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings: prompt eval time = 14309.46 ms /   512 tokens (   27.95 ms per token)
llama_print_timings:        eval time =   821.64 ms /     1 runs   (  821.64 ms per run)
llama_print_timings:       total time = 15986.28 ms

llama_print_timings:        load time = 15460.01 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings: prompt eval time = 14507.19 ms /   512 tokens (   28.33 ms per token)
llama_print_timings:        eval time =   900.15 ms /     1 runs   (  900.15 ms per run)
llama_print_timings:       total time = 16360.18 ms

For some reason, it seems like no matter what model I use, the response time stays around the same (Between 18 and 25 minutes). I did a series of test tonight and all of them returned around the same execution time.
I did some tests as well using llama.cpp, and it doesn't seem to change either, llama.cpp takes a long time to respond on my computer as well.
Do any of you have an idea of what I am doing wrong? Thanks.

Update:
llama.cpp took almost 2.5 hours to respond to my prompt, while the python binding took 18 minutes. I'm not sure if the problem is with the library, but rather with my computer.
Maybe my computer has something to do with the slow response time. If any of you have an idea as to why it is so slow on my PC, I'd love to hear it.

With LLaMA.cpp:
llama_print_timings:        load time =  3927.93 ms
llama_print_timings:      sample time =     5.51 ms /    10 runs   (    0.55 ms per run)
llama_print_timings: prompt eval time = 21902.09 ms /    48 tokens (  456.29 ms per token)
llama_print_timings:        eval time =  4735.28 ms /     9 runs   (  526.14 ms per run)
llama_print_timings:       total time = 27363.51 ms

With llama-cpp-python:
Output generated in 121.29 seconds (0.06 tokens/s, 7 tokens, context 44)

It is significantly slower on orders of magnitude

With LLaMA.cpp:
llama_print_timings:        load time =  3927.93 ms
llama_print_timings:      sample time =     5.51 ms /    10 runs   (    0.55 ms per run)
llama_print_timings: prompt eval time = 21902.09 ms /    48 tokens (  456.29 ms per token)
llama_print_timings:        eval time =  4735.28 ms /     9 runs   (  526.14 ms per run)
llama_print_timings:       total time = 27363.51 ms

With llama-cpp-python:
Output generated in 121.29 seconds (0.06 tokens/s, 7 tokens, context 44)

It is significantly slower on orders of magnitude

I see, although it shouldn't take around 18 minutes to generate an output, should it?

It shouldn't. Make sure you have the latest version of llama.cpp installed and make -j. They might have added some optimizations. Also make sure you have enough memory. It falls back to swap which is very slow

I see, I'll try that and I'll keep you updated. By the way, is there any way to check how much memory is used by the binding during its execution?

System monitor / task manager and look for python processes

I'm still not sure why this python binding is so much slower than the standard llama.cpp though

I'm not sure either. After some checks, it seems like my python script using the binding goes up to around (more or less) 7.65 Gb of memory used in average. Just in case, I updated the library as well, since I noticed the owner of the repo did some editing. However, it still takes a lot of time to generate an answer to me prompt.
I have one running right now, I've launched it around 5 minutes ago, I'll keep you updated when it ends and how long it lasted.

@acheong08 I'm not sure either. After some checks, it seems like my python script using the binding goes up to around (more or less) 7.65 Gb of memory used in average. Just in case, I updated the library as well, since I noticed the owner of the repo did some editing. However, it still takes a lot of time to generate an answer to me prompt.
I have one running right now, I've launched it around 5 minutes ago, I'll keep you updated when it ends and how long it lasted.

Took 74 minutes this time. I really wonder what could be wrong.

is llamacpp even supporting gpu? I don't see any function or something to move things to different hardware just as in pytorch etc usually?

is llamacpp even supporting gpu? I don't see any function or something to move things to different hardware just as in pytorch etc usually?

no

Hi all,
I also confirm there's a performance issue with this, but I looks to me the problem is only inside the high_level (fastapi) variant.
The regular llama.cpp works fine on my server - it gives response to a simple instructions in a matter of seconds (it's a small soho server with 4 core i5 and a 16GB of memory). I also tried the low_level variant of this python binding and - interestingly enough - it works almost the same as llama.cpp (starts generating words in around ~5-6 seconds)!
When I finally tried with the web server version (high_level variant) I got much worse performance - usually takes around 25 seconds to start generating anything :( I started tinkering around with the parameters and increased the number of threads from 2 to 4 and got somewhat better results (wait time is now around 16 seconds more or less), but still like 2x slower than llama.cpp variant. I also tried to enable mmap (had to increase the size using ulimit command), but no luck - still the same (except the memory is now almost full and the swap starts eating the space). I even tried playing with other parameters, but it seems nothing can speed up this server - which is so weird since it's supposed to be a web service extension over a python wrapper (right?).
Didn't have time to dive deeper into the code, but my naive guess is that it looks like something is fishy in the high_level code that makes the generation slower than the low_level variant, but no idea why.
Let me know guys if you need some other info, I'm very interested in getting this to work since we love llms & python!
Cheers

@bratislav I'll definitely look into this, can you provide some minimal examples? Also there are some high level api examples do you mind confirming the issue is there or in the server. There may be some weird asyncio bugs with the server so just trying to figure out where this could be coming from.
I also provided a small example in this graph to show how you can generate a flamegraph to profile where the majority of time is being spent. That might help us track down wether it's a python issue or comes from different parameters / settings in llama.cpp https://github.com/abetlen/llama-cpp-python/discussions/51#discussioncomment-5562523

Here are my runs for the high level vs. low level api on my machine just for reference. I've highlighted the portions which are running in C++ in the brown and provided the system information and per token timings for reference.
Low Level API

High Level API

Update:
llama.cpp took almost 2.5 hours to respond to my prompt, while the python binding took 18 minutes. I'm not sure if the problem is with the library, but rather with my computer.
Maybe my computer has something to do with the slow response time. If any of you have an idea as to why it is so slow on my PC, I'd love to hear it.

Look at your memory and disk utilization. If your ram is anywhere near 100% you're going to have bad performance if it's trying to use swap to compensate for a lack of memory. If your disc utilization goes up as you run the executable, that's a sign that bad things are happening.
If it takes minutes to compute a response then something is definitely wrong with how things are running.

Hi @abetlen, thanks for a quick response.
Here's our soho server info:
Ubuntu 20.04.5 LTS
i5
16GB RAM
Python: 3.8.10
gcc: 9.4.0
I have downloaded the alpaca and the llama models from the torrent found here, using magnet link at the top of the page...not sure if the models are the right ones, but here are the files I'm using:
llama-7b-ggml-q4_0/ggml-model-q4_0.bin
alpaca-7b-ggml-q4_0-lora-merged/ggml-model-q4_0.bin
(haven't converted or modified them, I'm just running them)
Okay, so the first run is the regular (latest) llama.cpp, downloaded and built yesterday:
baki@srv-mini:~/llama.cpp$ ./examples/alpaca.sh
main: seed =<PHONE_NUMBER>
llama_model_load: loading model from './models/7B/ggml-alpaca-7b-q4.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: type    = 1
llama_model_load: ggml map size = 4017.70 MB
llama_model_load: ggml ctx size =  81.25 KB
llama_model_load: mem required  = 5809.78 MB (+ 1026.00 MB per state)
llama_model_load: loading tensors from './models/7B/ggml-alpaca-7b-q4.bin'
llama_model_load: model size =  4017.27 MB / num tensors = 291
llama_init_from_file: kv self size  =  256.00 MB

system_info: n_threads = 3 / 4 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
main: interactive mode on.
Reverse prompt: '### Instruction:

'
sampling: temp = 0.200000, top_k = 10000, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.000000
generate: n_ctx = 512, n_batch = 256, n_predict = 128, n_keep = 21

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - If you want to submit another line, end your input in '\'.

 Below is an instruction that describes a task. Write a response that appropriately completes the request.
> Hi, from now on your name is Bob, a virtual assistant made by Baki.
Hello, I'm Bob.

### Instruction:

> Who made you?
I was made by Baki.

### Instruction:

>

Timings:
Model loading (until first input shows): ~ 6 seconds
After first instruction, response shows after: ~7 seconds
After second instruction,  response shows after: ~4 seconds
Second run, I try the low_level python wrapper around the same llama.cpp version (downloaded into /vendor dir), on the same machine:
baki@srv-mini:~/llama-cpp-python$ python3 examples/low_level_api/low_level_api_chat_cpp.py --model ~/llama.cpp/models/7B/ggml-llama-7b-q4.bin -r "User:" --interactive
seed =<PHONE_NUMBER>
llama.cpp: loading model from /home/baki/llama.cpp/models/7B/ggml-llama-7b-q4.bin
llama_model_load_internal: format     = ggjt v1 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: f16        = 2
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =  59.11 KB
llama_model_load_internal: mem required  = 5809.32 MB (+ 2052.00 MB per state)
llama_init_from_file: kv self size  =  512.00 MB

system_info: n_threads = 4 / 4 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
interactive mode on.
Reverse prompt: 'User:'
sampling: temp = 0.8,top_k = 40,top_p = 0.95,repeat_last_n = 64,repeat_penalty = 1.1

generate: n_ctx = 512, n_batch = 8, n_predict = 128, n_keep = 0

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - If you want to submit another line, end your input in '\'.

 2014-15 Women's Lacrosse News
Virginia Tech Announces Schedule for its Second Season of Lax
BLACKSBURG, Va. – Virginia Tech women’s lacrosse head coach Shannon Smith announced her team’s 2015 schedule on Thursday morning (December 4). The Hokies open their season at home against Longwood (Jan. 29) before heading to North Carolina for a Feb. 1 matchup with Duke.
Six Women's Lacrosse Players Named All

Not sure if I messed up something with the params, but this example in interactive mode doesn't provide any input, it just starts spitting out words - and stops after some number. Also, if I change the high_level example from chat to llama, it just segfault:
baki@srv-mini:~/llama-cpp-python$ python3 examples/low_level_api/low_level_api_llama_cpp.py --model ~/llama.cpp/models/7B/ggml-llama-7b-q4.bin
error loading model: failed to open ../models/7B/ggml-model.bin: No such file or directory
llama_init_from_file: failed to load model
Segmentation fault (core dumped)

Timings of chat example:
Model loading (until first word shows):  almost immediately
Number of words per second: around 3 words/sec (Awesomee!)
Third run, I try with high_level_api example, as you requested:
baki@srv-mini:~/llama-cpp-python$ python3 examples/high_level_api/high_level_api_inference.py --model ~/llama.cpp/models/7B/ggml-llama-7b-q4.bin
llama.cpp: loading model from /home/baki/llama.cpp/models/7B/ggml-llama-7b-q4.bin
llama_model_load_internal: format     = ggjt v1 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: f16        = 2
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =  59.11 KB
llama_model_load_internal: mem required  = 5809.32 MB (+ 2052.00 MB per state)
llama_init_from_file: kv self size  =  512.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

llama_print_timings:        load time =  2471.73 ms
llama_print_timings:      sample time =    42.13 ms /    48 runs   (    0.88 ms per run)
llama_print_timings: prompt eval time =  5722.36 ms /    19 tokens (  301.18 ms per token)
llama_print_timings:        eval time = 17361.61 ms /    47 runs   (  369.40 ms per run)
llama_print_timings:       total time = 23131.09 ms
{
  "id": "cmpl-c6a253f3-102e-4eec-b888-a9442690e47f",
  "object": "text_completion",
  "created":<PHONE_NUMBER>,
  "model": "/home/baki/llama.cpp/models/7B/ggml-llama-7b-q4.bin",
  "choices": [
    {
      "text": "Question: What are the names of the planets in the solar system? Answer: 1. Mercury 2. Venus 3. Earth 4. Mars 5. Jupiter 6. Saturn 7. Uranus 8. Neptune 9. Pluto 10.",
      "index": 0,
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 19,
    "completion_tokens": 48,
    "total_tokens": 67
  }
}

Timings:
This took around 25 seconds! (+ subsequent runs don't speed things up, I always get the similar time!)
In a final run, I try the llama-cpp-python from the pip, firing up instructions from the /docs web client (v1/chat/completion endpoint with the default JSON request) - first instruction response shows after around the same time (25 seconds) and if I bump up the threads from 2 to 4 I get the response at around 16 seconds. Almost everything I've tried so far - cannot speed this up. Also, when I build python wheel from the source - the timings stays the same.
I can run all these examples on our production server,  which is a 4 Xeon IBM beast with a 128GB of memory to see if there's any difference.
TODO:

I haven't checked the SHA256 checksums of the model files yet (I hope the files are okay)

@abetlen Here are the outputs of the py-spy:
Low-level:
baki@srv-mini:~/llama-cpp-python$ py-spy record --native -o profile.svg -- python3 examples/low_level_api/low_level_api_llama_cpp.py -m ~/llama.cpp/models/7B/ggml-llama-7b-q4.bin
py-spy> Sampling process 100 times a second. Press Control-C to exit.

llama.cpp: loading model from /home/baki/llama.cpp/models/7B/ggml-llama-7b-q4.bin
llama_model_load_internal: format     = ggjt v1 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: f16        = 2
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =  59.11 KB
llama_model_load_internal: mem required  = 5809.32 MB (+ 2052.00 MB per state)
py-spy> 1.15s behind in sampling, results may be inaccurate. Try reducing the sampling rate
llama_init_from_file: kv self size  =  512.00 MB

### Instruction:
What is the capital of France?

### Response:
Paris.

### Questions:
Who was the first president of America?

llama_print_timings:        load time =  2942.98 ms
llama_print_timings:      sample time =    19.85 ms /    20 runs   (    0.99 ms per run)
llama_print_timings: prompt eval time =  6159.05 ms /    28 tokens (  219.97 ms per token)
llama_print_timings:        eval time =  4763.31 ms /    19 runs   (  250.70 ms per run)
llama_print_timings:       total time = 12137.12 ms

py-spy> Stopped sampling because process exited
py-spy> Wrote flamegraph data to 'profile.svg'. Samples: 1250 Errors: 0

Here is for high_level API:
baki@srv-mini:~/llama-cpp-python$ py-spy record --native -o profile2.svg -- python3 examples/high_level_api/high_level_api_inference.py -m ~/llama.cpp/models/7B/ggml-llama-7b-q4.bin
py-spy> Sampling process 100 times a second. Press Control-C to exit.

llama.cpp: loading model from /home/baki/llama.cpp/models/7B/ggml-llama-7b-q4.bin
llama_model_load_internal: format     = ggjt v1 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: f16        = 2
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =  59.11 KB
llama_model_load_internal: mem required  = 5809.32 MB (+ 2052.00 MB per state)
llama_init_from_file: kv self size  =  512.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

llama_print_timings:        load time =  2629.82 ms
llama_print_timings:      sample time =    45.66 ms /    48 runs   (    0.95 ms per run)
llama_print_timings: prompt eval time =  6087.97 ms /    19 tokens (  320.42 ms per token)
llama_print_timings:        eval time = 18470.45 ms /    47 runs   (  392.99 ms per run)
llama_print_timings:       total time = 24609.50 ms
{
  "id": "cmpl-94bf3241-852a-469e-8710-fd4702e280d4",
  "object": "text_completion",
  "created":<PHONE_NUMBER>,
  "model": "/home/baki/llama.cpp/models/7B/ggml-llama-7b-q4.bin",
  "choices": [
    {
      "text": "Question: What are the names of the planets in the solar system? Answer: 1. Mercury 2. Venus 3. Earth 4. Mars 5. Jupiter 6. Saturn 7. Uranus 8. Neptune 9. Pluto 10.",
      "index": 0,
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 19,
    "completion_tokens": 48,
    "total_tokens": 67
  }
}

Latest version fixed my issues

Difference seems to be minimal now

It seems like it is to me as well. It now takes just a few seconds instead of those 18 minutes I had before. I don't know what happened but I am glad it got fixed. Thanks!

It might have just been a regression in the base llama.cpp library but who knows

@abetlen  @MartinPJB @acheong08  Guys, I've updated both llama.cpp and llama-cpp-python and I'm still getting the same timings as before - no change whatsoever :( I'm starting to think my model is wrong, are you guys willing to share the link from where you downloaded the .bin files (or at least sha256 sum so I can compare with mine)?
Did you just created the 8.bin file from weights or you downloaded already-made file in ggml? Are you using ggml v2 maybe?
I've tried with another alpaca bin file I found on the web, and the inference in high_level example is even slower - 44 seconds :(
Thanks

@bratislav The issue was not with the bin files. I tested with the same model throughout. Specifically https://huggingface.co/Pi3141/gpt4-x-alpaca-native-13B-ggml

make sure to cd vendor && rm -rf llama.cpp && git clone https://github.com/ggerganov/llama.cpp/ && cd .. && pip3 install .

this fetches you the latest version

Latest version fixed my issues

Difference seems to be minimal now

It might have just been a regression in the base llama.cpp library but who knows

Unsure if the slowness here was related, but there was a bunch of performance regression stuff in the base lib that I was vaguely following that was resolved recently:

Thank you for this hard work - I missed this regression because I rarely run generations with more than a few tens of tokens. The problem is that the transpose operation for the V matrix is very slow and becomes slower and slower with every new token added.
I think I have provided a fix here: https://github.com/ggerganov/llama.cpp/pull/775
Tested only on M1 so far
Originally posted by @ggerganov in https://github.com/ggerganov/llama.cpp/issues/603#issuecomment-1497569526

https://github.com/ggerganov/llama.cpp/issues/#775 was merged pretty quickly, cyyynthia if it's not too much trouble, could you update your pretty graphs with that?

@sw The graphs aren't super pretty this time because I didn't take the time to properly close everything and had a bunch of things open in background while the test was running 😅
That being said, the regression appears to be gone. 🎉  Here's the graphs and the raw CSV:

token_times.csv
Originally posted by @cyyynthia in https://github.com/ggerganov/llama.cpp/issues/603#issuecomment-1498024558

Well I guess that settles it. The Great Wizard Georgi has saved the day! Thanks to @cyyynthia and @KASR for putting in the hard work of tracking this down.
I have opened https://github.com/ggerganov/llama.cpp/issues/#790 to track the discrepancy in the different partial times vs total time. I think this issue could be closed. Thanks everyone.
Originally posted by @sw in https://github.com/ggerganov/llama.cpp/issues/603#issuecomment-1498071071


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/74374

Triggering TagBot for merged registry pull request
This extra notification is being sent because I expected a tag to exist by now, but it doesn't.
You may want to check your TagBot configuration to ensure that it's running, and if it is, check the logs to make sure that there are no errors.

Triggering TagBot for merged registry pull request
This extra notification is being sent because I expected a tag to exist by now, but it doesn't.
You may want to check your TagBot configuration to ensure that it's running, and if it is, check the logs to make sure that there are no errors.

Triggering TagBot for merged registry pull request
This extra notification is being sent because I expected a tag to exist by now, but it doesn't.
You may want to check your TagBot configuration to ensure that it's running, and if it is, check the logs to make sure that there are no errors.

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/78043

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/84894


unknown include #documentation

It includes lots of
<key>include</key>
<string>#escape</string>

Sublime will outputs dozen of unknown include #documentation

Sorry I have no idea what this means. Can you give a little more detail please?

startup logs:
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation

If I disable sublime-markdown-extended, then no logs.

And I don't know which one language cause it.

strange, I've never seen that before. I have no clue what might cause that in ST. anyone have suggestions?

Download the 3083 (I test it on winx64,should be same on other platform) from official.
Unzip it.
Install packagecontrol.
Close it.
Open again.
Install sublime-markdown-extended. You'll see it in the log.

DPI scale: 1.25
startup, version: 3083 windows x64 channel: stable
executable: /C/Users/yourname/Desktop/Sublime Text Build 3083 x64 - Copy/sublime_text.exe
working dir: /C/Users/yourname/Desktop/Sublime Text Build 3083 x64 - Copy
packages path: /C/Users/yourname/Desktop/Sublime Text Build 3083 x64 - Copy/Data/Packages
state path: /C/Users/yourname/Desktop/Sublime Text Build 3083 x64 - Copy/Data/Local
zip path: /C/Users/yourname/Desktop/Sublime Text Build 3083 x64 - Copy/Packages
zip path: /C/Users/yourname/Desktop/Sublime Text Build 3083 x64 - Copy/Data/Installed Packages
ignored_packages: ["Vintage"]
pre session restore time: 0.100613
startup time: 0.153613
first paint time: 0.154613
reloading plugin Default.block
reloading plugin Default.comment
reloading plugin Default.copy_path
reloading plugin Default.delete_word
reloading plugin Default.detect_indentation
reloading plugin Default.duplicate_line
reloading plugin Default.echo
reloading plugin Default.exec
reloading plugin Default.fold
reloading plugin Default.font
reloading plugin Default.goto_line
reloading plugin Default.history_list
reloading plugin Default.indentation
reloading plugin Default.kill_ring
reloading plugin Default.mark
reloading plugin Default.new_templates
reloading plugin Default.open_context_url
reloading plugin Default.open_file_settings
reloading plugin Default.open_in_browser
reloading plugin Default.pane
reloading plugin Default.paragraph
reloading plugin Default.paste_from_history
reloading plugin Default.quick_panel
reloading plugin Default.save_on_focus_lost
reloading plugin Default.scroll
reloading plugin Default.set_unsaved_view_name
reloading plugin Default.side_bar
reloading plugin Default.sort
reloading plugin Default.swap_line
reloading plugin Default.switch_file
reloading plugin Default.symbol
reloading plugin Default.transform
reloading plugin Default.transpose
reloading plugin Default.trim_trailing_white_space
reloading plugin CSS.css_completions
reloading plugin Diff.diff
reloading plugin HTML.encode_html_entities
reloading plugin HTML.html_completions
reloading plugin 0_package_control_loader.00-package_control
reloading plugin 0_package_control_loader.02-bz2
reloading plugin Package Control.bootstrap
reloading plugin Package Control.Package Control
plugins loaded
Package Control: Skipping automatic upgrade, last run at 2015-04-10 02:59:04, next run at 2015-04-10 03:59:04 or after
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation
unknown include #documentation

I can confirm that my console is littered w/ this.

it's entirely possible (maybe likely) that a conflict with another package causes this. would be great to find out more detail if anyone has any clues. I can't reproduce

I will try to find more info later on. The following is from my memory.
In the recent sublime text beta release, this happens only when opening some file . And I can't reproduce it in 3092. Maybe it is just a sublime bug. The recent release is all about syntax-file.

Closing due to age and since I've not been able to reproduce this (and we haven't had more reports about it)