[main] Stage 2 build is failing in prep step trying to find 9.0 P7

/vmr/eng/tools/BinaryToolKit/BinaryToolKit.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Ref with version (= 9.0.0-preview.7.24405.7)
/vmr/eng/tools/BinaryToolKit/BinaryToolKit.csproj : error NU1102:   - Found 8 version(s) in /tmp/tmp.RpNGwiWXMy [ Nearest version: 9.0.0-rc.1.24414.5 ]
/vmr/eng/tools/BinaryToolKit/BinaryToolKit.csproj : error NU1102:   - Found 0 version(s) in /vmr/.dotnet/library-packs
/vmr/eng/tools/BinaryToolKit/BinaryToolKit.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-x64. No packages exist with this id in source(s): /tmp/tmp.RpNGwiWXMy, /vmr/.dotnet/library-packs
/vmr/eng/tools/BinaryToolKit/BinaryToolKit.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Ref with version (= 9.0.0-preview.7.24406.2)
/vmr/eng/tools/BinaryToolKit/BinaryToolKit.csproj : error NU1102:   - Found 3 version(s) in /tmp/tmp.RpNGwiWXMy [ Nearest version: 9.0.0-rc.1.24414.4 ]
/vmr/eng/tools/BinaryToolKit/BinaryToolKit.csproj : error NU1102:   - Found 0 version(s) in /vmr/.dotnet/library-packs

Example build (internal link)
This is a regression caused by https://github.com/dotnet/sdk/pull/42969

@NikolaMilosavljevic - Can you investigate as part of SB Monitor?


💥 Implementation of Direction Catastrophe

Should this program cancel opposite directions which are not next to each other in the input list?
simplify_directions(['NORTH', 'EAST', 'SOUTH'])

returns
['NORTH', 'EAST', 'SOUTH']

Technically, it should cancel NORTH and SOUTH because we end up in the same position whether they are cancelled or not.

Should this program cancel opposite directions which are not next to each other in the input list?
simplify_directions(['NORTH', 'EAST', 'SOUTH'])

returns
['NORTH', 'EAST', 'SOUTH']

Technically, it should cancel NORTH and SOUTH because we end up in the same position whether they are cancelled or not.

If two consecutive directions are opposite then they should be cancelled. In this case ['NORTH', 'EAST', 'SOUTH'], the person  took North to go to (0,1) then went east (1,1) and lastly came to (1,0) by taking SOUTH.  As positions (0,1) and (1,0) are not equal, therefore we dont end up in the same position.

Should this program cancel opposite directions which are not next to each other in the input list?
simplify_directions(['NORTH', 'EAST', 'SOUTH'])

returns
['NORTH', 'EAST', 'SOUTH']

Technically, it should cancel NORTH and SOUTH because we end up in the same position whether they are cancelled or not.

If two consecutive directions are opposite then they should be cancelled. In this case ['NORTH', 'EAST', 'SOUTH'], the person took North to go to (0,1) then went east (1,1) and lastly came to (1,0) by taking SOUTH. As positions (0,1) and (1,0) are not equal, therefore we dont end up in the same position.

Let's break it down, step by step. If we don't cancel NORTH and SOUTH, it goes as following:

Starts at (0, 0)
NORTH (0, 1)
EAST (1, 1)
SOUTH (1, 0)
So START = (0, 0) and End = (1, 0)

Now, if we remove the opposite directions, it leaves us with ['EAST']. So it becomes:

Starts at (0, 0)
EAST (1, 0)
Finally, START = (0, 0) and End = (1, 0)

The starting and ending points of both runs are the same. So I believe we should be cancelling out non-consecutive opposite directions as well.

I understand your explanation technically. But in a real-world scenario, suppose there is no actual way to get to (1,0) from (0,0). The road doesn't directly go there or is blocked.
The example given in question statement also refers to only immediate opposite directions.


Update Github page (documentation) for MVI

https://www.gitbook.com seems to be a great tool for writing docs.
Moving markdown from jekyll to gitbook should be easy

How am I supposed to use the MVI part of this library if there is no documentation? Please add that as soon as possible, regardless of which tool you use!


IF YOU HAVE REQUEST ISSUES: Hackney 1.6.6 is broken

Downgrade to 1.6.5, or upgrade to 1.7.1

For anyone looking for more info on this, the error I got is as follows:
iex> ExAws.S3.put_object("test_bucket", "test.txt", "test") |> ExAws.request
{:error,
 {:http_error, 400,
  %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><Error>BadRequest<Message>An error occurred when parsing the HTTP request.</Message></Error>",
    headers: [{"Date", "Thu, 09 Mar 17 03:55:06 GMT"}, {"Connection", "close"},
     {"Transfer-Encoding", "chunked"},
     {"Content-Type", "application/xml"}], status_code: 400}}}


Automatically update the version in package.json to versionToPublish

Right now the version in client-js/package.json needs to be changed manually. This is an error-prone approach.
A suggestion is to update the package.json with new version during buildJs task.

Implemented in #42.


Rescue JSON decoding errors rather than killing WorkerManager

This is a first pass at addressing https://github.com/edgurgel/verk/issues/89. I was initially unsure of how to skip jobs because QueueManager.dequeuewill have already moved them to the inprogress queue. But calling QueueManager.ack will remove it.
It might be cleaner to add a new handler to QueueManager that does the same thing as :ack but is explicitly used for removal (whereas using :ack seems to imply removal of a successfully completed job).
It is pretty difficult to add a test for this as enqueuing jobs from Verk uses Poison encoding and thus the interoperability problem I am running into cannot be recreated easily.

Changes Unknown when pulling 04ecd4b9df6aaa46b292ed997310e09a14b4459f on keyan:decode_rescue_malformed into ** on edgurgel:master**.

Changes Unknown when pulling dee631f0e533ab92cb0c15730cbad5b3f1ed98c7 on keyan:decode_rescue_malformed into ** on edgurgel:master**.

Coverage increased (+0.5%) to 83.871% when pulling 19313a30ee8c4a56a0fcc9b40b8cfe2541fa885b on keyan:decode_rescue_malformed into ddd67707f65d2444115c460f56b932fa5b9a4a8e on edgurgel:master.

@edgurgel, ready for another look.
After taking into consideration both of your suggestions, I realized the diff was quite large, so I rebased. Also added some tests for the new Job.decode/1 and updated WorkerManager.handle_info/2 methods.

I just ran into an issue with this fix, when we try to remove the job from the inprogress queue by calling QueueManager.ack/2 the handle_cast callback attempts to access job.original_json, which raises:
18:36:32.178 [error] GenServer :"php_queue:emails.queue_manager" terminating
** (ArgumentError) argument error
    :erlang.apply("", :original_json, [])
    (verk) Verk.QueueManager.handle_cast/2
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:ack, ""}}
State: %Verk.QueueManager.State{node_id: "1", queue_name: :"php_queue:emails", redis: #PID<0.5955.0>}

Does it make sense to have a new handler in QueueManager? Say:
def handle_cast({:malformed, job}, state) do
  case Redix.command(state.redis, ["LREM", inprogress(state.queue_name, state.node_id), "-1", job]) do
    {:ok, 1} -> :ok
    _ -> Logger.error("Failed to acknowledge job #{inspect job}")
  end
  {:noreply, state}
end

Coverage increased (+0.1%) to 83.467% when pulling 5c58e57b65327fbf8ce0c4ae6667db210a32dcb4 on keyan:decode_rescue_malformed into ddd67707f65d2444115c460f56b932fa5b9a4a8e on edgurgel:master.

Hey sorry for not replying but this week has been pretty busy. I'm keen to move this forward this weekend. Thank you again for pushing this forward!

No worries! Thanks for staying on top of this.
We've been running this branch in production for the last couple days without any issues. Of course I am happy to work on this further to make sure we can get a fix merged so we can stop using a fork in production 😄 .

LGTM! I will release a new version soon! 🎉


Fixed addCategoryCommand success message

Fixed addCategory success message to more accurate fit

LGTM. I like the fact that you can notice the effect between different messages under different conditions.

LGTM! What a minute detail you have noticed!


Add a root Makefile and setup.py to install everything easily

With those it become easy to package the project: https://git.sr.ht/~stacyharper/personal-aports/tree/main/item/stacy/celeste-magic-timer/APKBUILD

distutils is hilariously deprecated (removed in python 3.12!), can you swap it for setuptools?

Ah okay. Sorry my Python is a bit outdated :) Done!

I added a commit to enforce the modules to be installed in a dedicated folder, so that everything does't stay in the common site-packages/ folder. For this to works, I had to use relative imports for the module to importe themselves correctly.

I know this is just me being anal, but it's considered a faux pas to have capital letters in python package names. Can you name the package celeste_magic_timer?
This isn't something I think is a good idea for right now, but we may want to make the tracer build and distribute as part of the python package.

Merged. Thank you so much!


feat: updating .secrets.baseline to suppress false positives

dependency: none
PR summary
The PR's objective is to surppress the newly detected false positive secrets.
Fixes:
List of false positives:
FOUND SECRETS:
File: CHANGELOG.md Line: 150 Type: Secret Keyword
File: authenticatedoriginpullapiv1/authenticated_origin_pull_api_v1.go Line: 421 Type: Secret Keyword
File: authenticatedoriginpullapiv1/authenticated_origin_pull_api_v1.go Line: 762 Type: Secret Keyword
File: authenticatedoriginpullapiv1/authenticated_origin_pull_api_v1_integration_test.go Line: 168 Type: Private Key
File: authenticatedoriginpullapiv1/authenticated_origin_pull_api_v1_test.go Line: 66 Type: Secret Keyword
File: authenticatedoriginpullapiv1/authenticated_origin_pull_api_v1_test.go Line: 3056 Type: Private Key
File: botanalyticsv1/bot_analytics_v1_test.go Line: 66 Type: Secret Keyword
File: botmanagementv1/bot_management_v1_test.go Line: 66 Type: Secret Keyword
File: directlink.env.properties Line: 2 Type: Secret Keyword
File: dnssvcsv1/dns_svcs_v1_test.go Line: 14497 Type: Hex High Entropy String
File: mtlsv1/mtls_v1_test.go Line: 62 Type: Secret Keyword
File: mtlsv1/mtls_v1_test.go Line: 2249 Type: Hex High Entropy String
File: mtlsv1/mtls_v1_test.go Line: 2249 Type: Hex High Entropy String

PR Checklist
Please make sure that your PR fulfills the following requirements:

[ ] The commit message follows the Angular Commit Message Guidelines.
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] New tests
[x] Build/CI related changes
[ ] Documentation content changes
[ ] Other (please describe)

What is the current behavior?

What is the new behavior?

Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information

:tada: This PR is included in version 0.43.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Some of the notebooks such as tutorialML should be moved out of Fourier directory

Perhaps there is a need to reorganize notebook folders. Identify at the top of each notebook the concepts covered in the notebook.

Maybe remove .py files. They just provide the same content in another file which is hard to maintain.


[SPARK][SPARK-10370]Cancel all running attempts while that stage marked as finished

@squito  thank to you already add some code to prevent resubmitting failed stage while FetchFailedException was thrown by task in previous stage attempts. So I just remove that logic in my previous modification.

Merged build triggered.

Merged build started.

Test build #41867 has started for   PR 8550 at commit 75b9779.

Test build #41867 has finished for   PR 8550 at commit 75b9779.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41867/
Test FAILed.

Hi @suyanNone thanks for submitting this, but this is actually not what that jira was referring to.  We don't just need to cancel tasks that are already running -- we also need to mark the existing task set as a zombie in order to prevent even more tasks from getting scheduled.  Canceling the running tasks would also be good, but it may be done as part of SPARK-2666, and there is still a bit of design discussion needed on that one.  For SPARK-10370, I think we're blocked by SPARK-10372 to have a good way to test it.
Unfortunately these small changes look simple but are tough to reason about, so I think we want to proceed very cautiously.  Do you mind closing this pr?

@squito yean, I will close that

@squito  I close that patch, because there have some errors in test.
call taskScheduler.cancelTasks(stage.id, true) in MarkStageAsFinished, It will mark all the no usefull tasksets related to that stage as zombie, because we already mark it as finished, and no need to deal with any running task for that stage. So first that prevent task from scheduling, second
it may be good to kill the running task for already finished stage, the task may run a long time.

For Spark-2666, this patch is aim to cancel all stage tasks as long as a FetchFailedException was thrown, and I think is no related to this patch, right? because this patch is just cancel the task which his stage is marked as finished.

@suyanNone ah, you are right, sorry I didn't see that taskScheduler.cancelTasks marks the TaskSetManager as zombie as well.  so it would have the desired effect of marking the stage as zombie, but it has some strange side-effects, including logging that the stage has failed and sending a TaskSetFailed event to the dagscheduler, though in fact nothing has failed, the rest of the tasks are just no longer necessary.
That was the original purpose of SPARK-2666 ... I am proposing widening it a bit, to cover the interaction between "zombie" tasksets and cancelling tasks.  I think that will lead to a net improvement in the scheduler code.  But its possible they should just be completely separate issues, I think we need more input before deciding that.
Can we move this discussion to the jira for SPARK-2666?  You have made some good points and they will most likely get lost on this pr ...


Add revoke key feature to wskadmin

A small enhancement to wskadmin to revoke a key. This already exists in wskadmin-next (#4005) but recently has an experience where running the new tool was too slow and took several seconds (it was not a very powerful CPU) and running the python version was faster. I could add a test for this reusing what's there for wskadmin-next.
@chetanmeh thoughts?
(I also added a "generate only" feature to get a new uuid:key without actually storing them, was handy for testing without affecting the database. I can add this to -next.)

Description

Related issue and scope

[ ] I opened an issue to propose and discuss this change (#????)

My changes affect the following components

[ ] API
[ ] Controller
[ ] Message Bus (e.g., Kafka)
[ ] Loadbalancer
[ ] Invoker
[ ] Intrinsic actions (e.g., sequences, conductors)
[ ] Data stores (e.g., CouchDB)
[ ] Tests
[ ] Deployment
[ ] CLI
[x] General tooling
[ ] Documentation

Types of changes

[ ] Bug fix (generally a non-breaking change which closes an issue).
[ ] Enhancement or new feature (adds new functionality).
[ ] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

[x] I signed an Apache CLA.
[x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[ ] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[ ] I updated the documentation where necessary.

Codecov Report

Merging #4249 into master will decrease coverage by 4.75%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4249      +/-   ##
==========================================
- Coverage   86.07%   81.31%   -4.76%
==========================================
  Files         156      156
  Lines        7481     7481
  Branches      480      480
==========================================
- Hits         6439     6083     -356
- Misses       1042     1398     +356

Impacted Files
Coverage Δ

...core/database/cosmosdb/RxObservableImplicits.scala
0% <0%> (-100%)
:arrow_down:

...core/database/cosmosdb/CosmosDBArtifactStore.scala
0% <0%> (-95.6%)
:arrow_down:

...sk/core/database/cosmosdb/CosmosDBViewMapper.scala
0% <0%> (-92.6%)
:arrow_down:

...whisk/core/database/cosmosdb/CosmosDBSupport.scala
0% <0%> (-83.34%)
:arrow_down:

...abase/cosmosdb/CosmosDBArtifactStoreProvider.scala
4.54% <0%> (-45.46%)
:arrow_down:

...in/scala/org/apache/openwhisk/common/Counter.scala
40% <0%> (-20%)
:arrow_down:

...nwhisk/core/database/cosmosdb/CosmosDBConfig.scala
93.75% <0%> (-6.25%)
:arrow_down:

...rg/apache/openwhisk/common/ForcibleSemaphore.scala
92.3% <0%> (+3.84%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update abe2d33...5672528. Read the comment docs.

For reference here is the updated argument list
wskadmin user create -h
usage: wskadmin user create [-h] [-u AUTH] [-ns NAMESPACE] [-r] [-g] [-s]
                            subject

positional arguments:
  subject               the subject to create

optional arguments:
  -h, --help            show this help message and exit
  -u AUTH, --auth AUTH  the uuid:key to initialize the subject authorization
                        key with
  -ns NAMESPACE, --namespace NAMESPACE
                        create key for given namespace instead (defaults to
                        subject id
  -r, --revoke          revoke existing key and create a new one
  -g, --genonly         generate a uuid and key but do not store them in the
                        database
  -s, --silent          do not should the new key on the console


Varying Json response of the APIs

API

https://api.covid19india.org/raw_data.json
https://api.covid19india.org/raw_data1.json
https://api.covid19india.org/raw_data2.json
https://api.covid19india.org/raw_data3.json

Issue
Many applications are using the raw data API to build COVID-19  statistical dashboards and content.The https://api.covid19india.org/raw_data.json (till 26 April) provides unique patient number in patientnumber key.While the Live API does not provide any unique patient number to identity and compare it to prior records.(Only provides the entry ID).
Solution

Please maintain a consistent response of the API.
Please provide a unique identifier for the patients.

The data curation process is changing rapidly to adapt to new scenarios. Hence the change in structure in raw_data3. We suggest using the aggregate APIs provided for any live application. That will have more stability because our website itself is using those endpoints.

Please maintain a consistent response of the API.
We will try to wrangle all the files together and provide it in the format of raw_data3 soon. However, there would be mismatches, as states move around the patient numbers. History may not reflect the current situation.

Please provide a unique identifier for the patients.
We used to do this, however, we have now shifted to aggregate entry for states which are just reporting districtwise numbers (or sometime, just a single number for the state).

If you have further questions regarding raw_data, please do follow the discussion here :https://github.com/covid19india/api/issues/195
Thank you!


Question: installation and github app usage

configs
from conf.settings import settings
from githubkit import GitHub, AppInstallationAuthStrategy
from githubkit.retry import RetryChainDecision, RetryRateLimit, RetryServerError
with open('./static/private-key.pem', 'rb') as pem_file:
    private_key = pem_file.read()
github = GitHub(
    AppInstallationAuthStrategy(
        app_id=settings.GITHUB_APP_ID,
        private_key=private_key,
        installation_id=settings.GITHUB_APP_INSTALLATION_ID,
        client_id=settings.GITHUB_APP_CLIENT_ID,
        client_secret=settings.GITHUB_APP_CLIENT_SECRET
    ),
    base_url="https://api.github.com/",
    accept_format="full+json",
    previews=["starfox"],
    user_agent="GitHubKit/Python",
    follow_redirects=True,
    timeout=None,
    http_cache=True,
    auto_retry=RetryChainDecision(RetryRateLimit(max_retry=1), RetryServerError(max_retry=1)),
)

API call
response = await github.rest("2022-11-28").repos.async_get(owner="My_organization", repo="MyRepository")

error
backend  | 2024-05-24 18:23:43,000 - INFO - httptools_impl: <IP_ADDRESS>:55210 - "GET /v1/github/graphql/ HTTP/1.1" 500
backend  | ERROR:    Exception in ASGI application
backend  | Traceback (most recent call last):
backend  |   File "/usr/local/lib/python3.12/site-packages/githubkit/core.py", line 306, in _arequest
backend  |     return await client.request(
backend  |            ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1574, in request
backend  |     return await self.send(request, auth=auth, follow_redirects=follow_redirects)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1661, in send
backend  |     response = await self._send_handling_auth(
backend  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1686, in _send_handling_auth
backend  |     request = await auth_flow.__anext__()
backend  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/githubkit/auth/app.py", line 233, in async_auth_flow
backend  |     token_request.headers["Authorization"] = f"Bearer {await self.aget_jwt()}"
backend  |                                                        ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/githubkit/auth/app.py", line 89, in aget_jwt
backend  |     token = self._create_jwt()
backend  |             ^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/githubkit/auth/app.py", line 70, in _create_jwt
backend  |     return jwt.encode(
backend  |            ^^^^^^^^^^
backend  | AttributeError: module 'jwt' has no attribute 'encode'
backend  |
backend  | The above exception was the direct cause of the following exception:
backend  |
backend  | Traceback (most recent call last):
backend  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
backend  |     result = await app(  # type: ignore[func-returns-value]
backend  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
backend  |     return await self.app(scope, receive, send)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
backend  |     await super().__call__(scope, receive, send)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 123, in __call__
backend  |     await self.middleware_stack(scope, receive, send)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
backend  |     raise exc
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
backend  |     await self.app(scope, receive, _send)
backend  |   File "/usr/local/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 174, in __call__
backend  |     raise exc
backend  |   File "/usr/local/lib/python3.12/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 172, in __call__
backend  |     await self.app(scope, receive, send_wrapper)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
backend  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
backend  |     raise exc
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
backend  |     await app(scope, receive, sender)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 756, in __call__
backend  |     await self.middleware_stack(scope, receive, send)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 776, in app
backend  |     await route.handle(scope, receive, send)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 297, in handle
backend  |     await self.app(scope, receive, send)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 77, in app
backend  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
backend  |     raise exc
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
backend  |     await app(scope, receive, sender)
backend  |   File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 72, in app
backend  |     response = await func(request)
backend  |                ^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app
backend  |     raw_response = await run_endpoint_function(
backend  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
backend  |     return await dependant.call(**values)
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/app/apps/github/urls.py", line 65, in success
backend  |     save = await EventHandler.process_event()
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/app/apps/github/controller.py", line 10, in process_event
backend  |     response = await github.rest("2022-11-28").repos.async_get(owner="BarakahApp", repo="gitops")
backend  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/githubkit/versions/v2022_11_28/rest/repos.py", line 1232, in async_get
backend  |     return await self._github.arequest(
backend  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/githubkit/core.py", line 548, in arequest
backend  |     raw_resp = await self._arequest(
backend  |                ^^^^^^^^^^^^^^^^^^^^^
backend  |   File "/usr/local/lib/python3.12/site-packages/githubkit/core.py", line 320, in _arequest
backend  |     raise RequestError(repr(e)) from e
backend  | githubkit.exception.RequestError: AttributeError("module 'jwt' has no attribute 'encode'")

it seems to be that I have jwt package but this kit requires PyJWT
[tool.poetry.dependencies]
PyJWT = { version = "^2.4.0", extras = ["crypto"], optional = true }

Yet, I don't think this is a problematic since I tried it with tokenStrategy and it worked.
But when I install PyJWT, error is different and I am working to fix it on my end. Just thought to bring this to your attention
githubkit.exception.RequestError: RequestFailed(method=POST, url=https://api.github.com/app/installations/123456789/access_tokens, status_code=404 Not Found)

Appreciate your guidance.
In addition, I am pretty confused about the documentation. The call I am making to get repositories was a sample I found it online. From where can I find reference of the functionality ?
for example if I want to get  a pull request comment or issue or action or anything ? and creating pull request or merge them and so on.
Finally, I am planning to use GraphQL. Hence, I choose this package and your effort are really amazing and appreciated. Appreciate a guidance or that as well.
How can I make a GraphQL call, validate the response and create commit via mutation using GraphQL

I just figured out the 404 error and was due having wrong installation ID for different app
Yet, for JWT package got resolved only after I change dependency from having jwt to PyJWT

githubkit requires the PyJWT package to work with GitHub APP. Please install the requirements with pip install githubkit[auth-app], as described in the docs. You do not need to install the pyjwt package by your self. If you follow the installation guide, the correct package will be installed automatically.

From where can I find reference of the functionality? for example if I want to get a pull request comment or issue or action or anything ? and creating pull request or merge them and so on.

You can find all the rest apis at the docs.github.com.
For example, if you want to get a issue, you can find the docs here. and you can work with githubkit like this: (if you use github app, sync example)
from githubkit import GitHub, App

# auth with your github app
g = GitHub(
    AppAuthStrategy(
        "<app_id>", "<private_key>"
    )
)

# get the repo installation first
resp = g.rest.apps.get_repo_installation("owner", "repo")
installation_id = resp.parsed_data.id

# then, switch to installation auth
g_repo = g.with_auth(g.auth.as_installation(installation_id))

# you can now get all repo info by the installation authed rest api
resp = g_repo.rest.issues.get("owner", "repo", issue_number)
issue = resp.parsed_data

all the auth flow mentioned above, you can find the docs at GitHub APP Guide. GitHub APP cannot get the repo data directly since the owner of the repo should install the app first. then, the installation_id will be created after the owner has installed the app. you can access the repo data now by auth with the installation_id.
Other apis to create a pull request or merge it also can be found at the github rest docs, like the pulls section.

How can I make a GraphQL call, validate the response and create commit via mutation using GraphQL

The GraphQL usage is very simple, the docs is here. You can first edit the graphql request body at the GitHub GraphQL Editor online then copy the graphql request to your code, send it, and the return value will be the response data. Note that, you still need to auth correctly like the rest example above.

Indeed I don't need to explicitly install PyJWT. Yet, I was using jwt package in a different module and since I installed jwt package explicitly and caused the mentioned error in GithubKit
When I removed jwt package and refactored my code to use PyJWT in the other module. GithubKit issue was resolved directly. That's why I am mentioning it here.
In addition, everything is working now in GithubKit
Finally, as you're referring to Github API as a documentation, which meant GithubKit is relying on Github schema only and not supporting functionalities on top of Github API. Correct me if I wrong
Will continue to build on GithubKit. Will get back to you if I faced any issue
Thanks for developing this amazing app

What you mentioned is that module name of the two package jwt and pyjwt are same. If you think this is an issue, please report to them. It's not the error with githubkit and githubkit can not prevent you to install the jwt.
githubkit is a GitHub SDK. you should read the github docs first.

If you have any question about githubkit, feel free to open an issue.


Docs: The output of the bring_your_own_formatter_from_scratch.py example does not match the documentation.

What were you searching in the docs?
In https://docs.powertools.aws.dev/lambda/python/latest/core/logger/#bring-your-own-formatter doc, the output is
{
    "event": "Collecting payment",
    "timestamp": "2021-05-03 11:47:12,494",
    "my_default_key": "test",
    "cold_start": true,
    "function_name": "test",
    "function_memory_size": 128,
    "function_arn": "arn:aws:lambda:eu-west-1:12345678910:function:test",
    "function_request_id": "52fdfc07-2182-154f-163f-5f0f9a621d72"
}

But actually, in my lambda environment the output is
{"event": "Collecting payment", "timestamp": null, "my_default_key": "test", "level": null, "location": null, "message": null, "service": "payment", "sampling_rate": null, "cold_start": true, "function_name": "PowertoolsTest", "function_memory_size": "128", "function_arn": "arn:aws:lambda:us-west-2:*******:function:PowertoolsTest", "function_request_id": "67a57243-9819-4637-af53-8b2116bd5dbd"}

timestamp and log level are null.
My lambda layer version is arn:aws:lambda:us-west-2:017000801446:layer:AWSLambdaPowertoolsPythonV2:48
Is this related to an existing documentation section?
No response
How can we improve?
Fix the wrong document
Got a suggestion in mind?
No response
Acknowledgment

[X] I understand the final update might be different from my proposed suggestion, or refused.

Thank you @wchaws for taking the time to report this - we'll look into it this afternoon.

Closing; please feel free to reopen if I missed anything when trying to reproduce.


Roll Flutter from f26dd3730df8 to 4fa91c949aae (15 revisions)

https://github.com/flutter/flutter/compare/f26dd3730df8...4fa91c949aae
2022-10-18<EMAIL_ADDRESS>Revert "[tool] ⚡ Install the corresponding APK in flutter run (#112768)" (flutter/flutter#113614)
2022-10-18<EMAIL_ADDRESS>[tool] ⚡ Install the corresponding APK in flutter run (flutter/flutter#112768)
2022-10-17<EMAIL_ADDRESS>Roll Flutter Engine from 2ced97978624 to 465ccd297406 (4 revisions) (flutter/flutter#113577)
2022-10-17<EMAIL_ADDRESS>Roll pub packages (flutter/flutter#113331)
2022-10-17<EMAIL_ADDRESS>Roll Flutter Engine from 2a6e1002eba6 to 2ced97978624 (4 revisions) (flutter/flutter#113568)
2022-10-17<EMAIL_ADDRESS>[Docs] Clarify that Flex does not wrap (flutter/flutter#113404)
2022-10-17<EMAIL_ADDRESS>Add support for Alt to CharacterActivator, add tests (flutter/flutter#113466)
2022-10-17<EMAIL_ADDRESS>Reland "Single tap on the previous selection should toggle the toolbar on iOS #108913" (flutter/flutter#111995)
2022-10-17<EMAIL_ADDRESS>Clean up _updateSelectionRects  (flutter/flutter#113425)
2022-10-17 mono Fix typo of PrimaryScrollController documentation (flutter/flutter#113361)
2022-10-17<EMAIL_ADDRESS>Roll Flutter Engine from 8b5324bf5cfd to 2a6e1002eba6 (3 revisions) (flutter/flutter#113558)
2022-10-17<EMAIL_ADDRESS>Show Xcode compilation errors at end of build, suppress stdout and stderr from Xcode (flutter/flutter#113302)
2022-10-17<EMAIL_ADDRESS>Roll Flutter Engine from 66e422c4a2fb to 8b5324bf5cfd (2 revisions) (flutter/flutter#113549)
2022-10-17<EMAIL_ADDRESS>Support the --no-devtools flag in "flutter run --machine" (flutter/flutter#113414)
2022-10-17<EMAIL_ADDRESS>[ci.yaml] Increase docs targets to use 32 cores (flutter/flutter#113465)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-plugins
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Plugins: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Improve how the status messages are being displayed

Move the status messages [eg:("Image sent", "Image received", "An error occurred")] to the middle of the screen.

@nsankethreddy i would like to work on could u tell me a little bit how the messages are displayed?

Hey @Sameer-Mann if you could run the server and client program and send images/files between the nodes. You will notice that the " image sent "," image received successfully " and all such similar messages  are just getting printed on the left side of the shell, so all I need you to do is move all the status messages to the centre of the shell.

ok got it


samples/bluetooth/ipsp: build failed for MICRO-BIT & NRF51-PCA10028 HW

I was trying to build the ipsp test from samples/bluetooth folder for the target MICRO-BIT & NRF51-PCA10028 HW.
Getting some build issue if I am trying to build for these respective targets. It seems SRAM space issue occurs.
Below is the error message appears on terminal.
[1/148] Generating always_rebuild
Building for board nrf51_pca10028
[144/148] Linking C executable zephyr/zephyr_prebuilt.elf
FAILED: zephyr/zephyr_prebuilt.elf
: && ccache sysroots/x86_64-pokysdk-linux/usr/bin/arm-zephyr-eabi/arm-zephyr-eabi-gcc    zephyr/CMakeFiles/zephyr_prebuilt.dir/misc/empty_file.c.obj  -o zephyr/zephyr_prebuilt.elf  -T zephyr/linker.cmd -Wl,-Map=/home/zephyr/samples/bluetooth/ipsp/build/zephyr/zephyr.map -u_OffsetAbsSyms -u_ConfigAbsSyms -e__start -Wl,--start-group -Wl,--whole-archive libapp.a zephyr/libzephyr.a zephyr/subsys/bluetooth/common/libsubsys__bluetooth__common.a zephyr/subsys/bluetooth/host/libsubsys__bluetooth__host.a zephyr/subsys/bluetooth/controller/libsubsys__bluetooth__controller.a zephyr/subsys/net/libsubsys__net.a zephyr/subsys/net/ip/libsubsys__net__ip.a zephyr/subsys/net/ip/l2/libsubsys__net__ip__l2.a zephyr/subsys/net/lib/app/libsubsys__net__lib__app.a -Wl,--no-whole-archive zephyr/kernel/libkernel.a zephyr/CMakeFiles/offsets.dir/arch/arm/core/offsets/offsets.c.obj -Wl,--end-group -L"/home/sysroots/armv5-zephyr-eabi/usr/lib/arm-zephyr-eabi/6.2.0/armv6-m" -L/home/zephyr/samples/bluetooth/ipsp/build/zephyr -lgcc -Wl,--print-memory-usage -nostartfiles -nodefaultlibs -nostdlib -static -no-pie -Wl,-X -Wl,-N -Wl,--gc-sections -Wl,--build-id=none && :
Memory region         Used Size  Region Size  %age Used
FLASH:      149004 B       256 KB     56.84%
SRAM:       39736 B        32 KB    121.26%
IDT_LIST:         132 B         2 KB      6.45
sysroots/x86_64-pokysdk-linux/usr/libexec/arm-zephyr-eabi/gcc/arm-zephyr-eabi/6.2.0/real-ld: zephyr/zephyr_prebuilt.elf section noinit' will not fit in region SRAM'
sysroots/x86_64-pokysdk-linux/usr/libexec/arm-zephyr-eabi/gcc/arm-zephyr-eabi/6.2.0/real-ld: section .intList VMA [0000000020008000,0000000020008083] overlaps section noinit VMA [00000000200023e0,0000000020008a67]
sysroots/x86_64-pokysdk-linux/usr/libexec/arm-zephyr-eabi/gcc/arm-zephyr-eabi/6.2.0/real-ld: region `SRAM' overflowed by 6968 bytes
collect2: error: ld returned 1 exit status
%
ninja: build stopped: subcommand failed.

It simply doesn't fit:
SRAM: 39736 B 32 KB 121.26%
There maybe possible to fine-tune the buffers sizes to make them fit on 32 KB, but given it is almost 7 KB over I think it will be pretty hard to squeeze.

closing, sample does not fit, reduce configs or the feature is just too big for the board.


IMU send maxes out at ~135Hz

A few things to try:

Use the xx_send_buf() function instead of the xx_send() function; it's supposed to be more efficient
Use the SCALED_IMU message type instead of HIGHRES_IMU; it's 22 bytes as opposed to 62
If that still doesn't work, create a custom mavlink message (e.g. SMALL_IMU) that strips out the timestamp (4 bytes) and maybe the magnetometer(?) (6 bytes), leaving us with 18 or 12 bytes.

Got it working at up to 500Hz (when nothing else is going on) by switching to SMALL_IMU message type.

That's fantastic!!! Nice job!


refactor shutdown sequence to use plugin.stop

Change shutdown sequence to be triggered by plugin.stop instead of ShutdownSignal exception.
Also remove any calls to: shutdown, finished, finished?, running? or terminating?
This depends on https://github.com/elastic/logstash/pull/3895

Fixed in #10


Add support for preserving line breaks in Comments

There ought to be an easier way to put line breaks in comments (other than manually editing the HTML :smiley:),
As written in Sketch.app:

As rendered in the export:

Then, in order to fix it, I have to find that section of the html file and add in line breaks like this:

added


Relocate accumulo-maven-plugin into separate repo

The accumulo-maven-plugin need not be released every time Accumulo is released. It can have its own release schedule, and should be independent of Accumulo itself.

Re-opening, because original repo still has maven-plugin directory.

New location at https://github.com/apache/accumulo-maven-plugin

Fix in #686


Link to the provisioning guide does not work

The link to the provisioning guide does not work and I can't find it elsewhere on customersource or partnersource. Can someone provide me with the guide?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0e859ad9-22e1-e14e-2c7a-6df3ba1482ca
Version Independent ID: 6eff91da-e5c3-3fcd-00f2-2fd690e7cc15
Content: Buy Finance and Operations (on-premises) - Finance & Operations | Dynamics 365 | #MSDyn365FO
Content Source: articles/fin-and-ops/get-started/purchase-on-premises.md
Service: dynamics-ax-applications
GitHub Login: @maertenm
Microsoft Alias: maertenm

@tino80, thanks for bringing this to our attention. We're working to resolve this issue and will let you know when a working link is available.
@ShellyBakke, please help to resolve this issue.

Bug 332477 has been logged to track this issue.

This is resolved now. Links are working.

The link provided in the documentation isn't working. There is a new link?

Which link is not working? I've tested all of them and cannot find any that do not work.


Remove TravisCI from project

Signed-off-by: Michael Peters<EMAIL_ADDRESS>

Replaces https://github.com/keylime/keylime/pull/540


Make coverall results advisory and not report as failure

Right now if test coverage decreases (even slightly) coveralls reports this as a failure and the PR is marked as having failures.
I find this distracting in that it makes it harder to see which PRs are ready to review/merge.
Can we make coveralls advisory so that it reports test coverage results to GitHub but doesn't report a failure if there is a decrease?

I tried but I have no access to the settings, you could update it in https://coveralls.io/github/kubeflow/tf-operator/settings.

I changed coverage decrease threshold for failure to 20%.
What is the coverage threshold for failure?

If we set coverage threshold to 20%, and our code coverage is 19%, then it will report a failure.
If we set coverage decrease threshold to 20%, and one PR decrease the coverage from 100% to 79%, then it will report a failure.
Then I think we could set all these two to 20%

I see, coverage threshold is blank right now. Lets see what happens.

Coveralls is no longer reporting failure see
https://github.com/kubeflow/tf-operator/pull/403#issuecomment-368725661

I reduced both COVERAGE THRESHOLD FOR FAILURE and COVERAGE DECREASE THRESHOLD FOR FAILURE to 0.0% and Coveralls Coverage no longer fails.


No SMTP servers were available for gmail.com

Hi@all
I have follow error:
_No SMTP servers were available for gmail.com. Tried gmail-smtp-in.l.google.com, alt1.gmail-smtp-in.l.google.com, alt2.gmail-smtp-in.l.google.com, alt3.gmail-smtp-in.l.google.com, and alt4.gmail-smtp-in.l.google.com
421 4.7.0 Try again later, closing connection._
Google said, postal use a wrong SSL encryption: https://support.google.com/mail/thread/40108790?hl=en
Does anybody hava a solution?

The most likely cause of this issue is your server provider blocking SMTP connections. Have a look at #564, #738, #1178, and #1233.
This tends to happen with newer accounts and there may be a way for you to request opening the SMTP ports from them.

@willpower232 thank you, for your comment. I asked my provider hetzner, they said: We don't have any outbound filter.
How i can test? I think it is a IPv6 problem.
root@postal ~ # telnet gmail-smtp-in.l.google.com 25
Trying 2a00:1450:400c:c0b::1a...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
421 4.7.0 Try again later, closing connection.
Connection closed by foreign host.

root@postal ~ # telnet <IP_ADDRESS> 25
Trying <IP_ADDRESS>...
Connected to <IP_ADDRESS>.
Escape character is '^]'.
220 mx.google.com ESMTP s134si13677047wme.134 - gsmtp

Is that possible to use only a IPv4 connection?

I believe it is possible, you can have a look at #813 and #435

@willpower232 thank you. My solution:
Disable ipv6 completely:
echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" >> /etc/sysctl.conf

Than i'am reboot the server. I checked the configuration with ifconfig. It looks good.
Than i check the telnet connection:
root@postal ~ # telnet gmail-smtp-in.l.google.com 25
Trying <IP_ADDRESS>...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP m18si23496243wrg.331 - gsmtp

Now postal works fine. I can not really explain where is error with gmail and IPv6.

Its not a perfect solution but glad it works.
I imagine that sending verified email with IPv6 is stricter than with IPv4 so hopefully it will get easier as everything slowly switches over.

Thanks, it solved the same problem for me.
But I don't understand why Google receives it on IPv6...after disabling it on Postal host...

Message for<EMAIL_ADDRESS>accepted by gmail-smtp-in.l.google.com (2a00:1450:4010:c08::1b)

You can verify that IPv6 is definitely off with the netstat command as you shouldn't see any udp6 or tcp6 lines.

I had to explicitly disable it for eth0
net.ipv6.conf.eth0.disable_ipv6 = 1 👍

Hi, I am still having this same issue.
Running the following gives:
simon@postal:~$ telnet gmail-smtp-in.l.google.com 25
Trying 2404:6800:4003:c11::1b...

But running the following gives:
simon@postal:~$ telnet <IP_ADDRESS> 25
Trying <IP_ADDRESS>...
Connected to <IP_ADDRESS>.
Escape character is '^]'.
220 mx.google.com ESMTP d3-20020adfe843000000b002c594da9230si8396343wrn.887 - gsmtp

My error message within postal is:
No SMTP servers were available for gmail.com. Tried gmail-smtp-in.l.google.com, alt1.gmail-smtp-in.l.google.com, alt2.gmail-smtp-in.l.google.com, alt3.gmail-smtp-in.l.google.com, and alt4.gmail-smtp-in.l.google.com

And I have made the following changes to /etc/sysctl.conf followed by a reboot:
# Copied from https://github.com/postalserver/postal/issues/1247#issuecomment-721290912
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1

From here (https://itsfoss.com/disable-ipv6-ubuntu-linux/)
I worked out my issue. I needed to follow the steps at that page:

If IPv6 is still enabled after rebooting, you must create (with root privileges) the file /etc/rc.local and fill it with:
#!/bin/bash
# /etc/rc.local

/etc/sysctl.d
/etc/init.d/procps restart

exit 0

Now use chmod command to make the file executable:
sudo chmod 755 /etc/rc.local

What this will do is manually read (during the boot time) the kernel parameters from your sysctl configuration file.


Polling doesn't stop completely when there is an error in receiveMessage

@TomFrost Bump :)

Oh man, totally missed this. Thanks! I have a couple tweaks before release but I'll grab 'em after merge.

Published in 0.7.0.

👍


Expression Tree Tutorial

I wrote two sections on this so far (rough draft version).
The work so far leads me to believe this should be broken up into several sections, due to the depth of the content.
I'd propose one article (page) for each of the top level headings in the current draft (including the placeholders).
Thoughts?
Is this moving in the right direction?

Yes, it looks good so far. I agree it makes sense to break it up - in general, if a conceptual topic runs more than 2-3 pages it tends to lose readers.

I'll keep working in that direction. I think it would make sense to make one pull request per article, as I break them up.

I agree that this could probably be broken up into a few sub-articles given the depth of the material.

I'm working on it. Please wait for a couple more revisions before the next review. I want to respect everyone's time, and what's there now isn't really ready for feedback.

I've now broken this up into 8 different articles. I'm adding the samples now,  Here are the articles that are part of this PR:  When I check each one, it's ready for review:
[ ] Overview (expression-trees.md)
[x] Explaining Expression Trees (expression-trees-explained.md)
[x] Expression Tree classes (expression-classes.md)
[ ] Executing Expression Trees (expression-trees-execution.md)
[ ] Interpreting Expression Trees (expression-trees-interpreting.md)
[ ] Building Expression Trees (expression-trees-building.md)
[ ] Translating Expression Trees (expression-trees-translating.md)
[ ] Summing up (expression-tress-summary.md)
One question I have as I keep building the samples: Where should those go? In this repository, or in the core-clr repository in the samples section?

One question I have as I keep building the samples: Where should those go? In this repository, or in the core-clr repository in the samples section?

This repo should have the samples.  The way we're viewing samples in this context is that they serve documentation.

Just finished proofreading, and updating all the associated samples (running on core).
This should be ready to review.

nice tutorial :clap:

Fixes #73

I'll get to reviewing this in the next few days.  It's a lot of content :+1:

@cartermp I agree. It ended up quite large. On the next project, I'm going to create sub-issues for each section once I do the outline. That should make it easier to review and merge smaller sections.

This is great! I learned a lot by reading it. A few things come to mind, as well as a short list of specific edits.

Add a link to the next article in the sequence at the end of each article? and maybe the previous ones, too, at the start of the topic.
Our audience is not necessarily familiar with Entity Framework and LINQ since a good part of the audience for .NET Core will be people who are coming from other platforms and haven't been using C# and following all the updates to it. I just think that starting with the three words "You know LINQ" would tend to say to readers who don't know LINQ that they shouldn't be reading this.

The next comments refer to some small edits in specific topics :
building:
line 72 "to create" not "to creates"
line 86 "follows"
line 137 remove "of an"
line 158 remove "of"
204 capitalize "Core" when referring to ".NET Core"
execution:
line 56 compile-time not "compile time"
161-162 compile-time
explained
line 60: structure - spelling
87 add "an" to "expression tree"
interpreting
8 straight-forward : remove hyphen
translating
6 undestand => understand
Thanks! Great info here. I'm excited to see more.

Went through it and left some feedback.  I'm a huge fan of the interpreting and translating sections.  Great work.

Updated based on all of @cartermp comments.

I think this is good enough to merge.  @stevehoag @ghogen @richlander @mairaw any further feedback?  If not, I'll merge by end of day today.

LGTM - I think you already hit the major points.

@cartermp I can review these more thoroughly later on but I've left some feedback

@mairaw sounds good, I'll hold off until after you've had a chance to review more thoroughly.

@cartermp I'm pretty swamped on some other deadlines, so I won't get to these soon. Don't hold off because of me

@mairaw okay.  I'll merge this in once @BillWagner addresses your current feedback.

@mairaw Wow. This was already a thorough review. You catch a lot.
@cartermp  I should be able to address of it during the day tomorrow.

Updated with all the feedback from @mairaw

:ship:ing :it:


[BUG] .npmrc is generated in all the packages

Describe the bug
With version 0.17.3 onward, projen is also generating .npmrc file in each directory based on https://github.com/sperka/aws-prototyping-sdk/blob/c85312c706e776ef49beed02fa97df3285759cf8/private/projects/pdk-monorepo-project.ts#L174
This lead to unnecessary file generation and warnings
npm WARN ignoring workspace config at /Users/rverma/dev/aws/nsa/packages/demo/.npmrc

More details of handling this is here https://github.com/npm/cli/issues/4605
Expected Behavior
To generate only file in the root package
Current Behavior
Generating npmrc in all the packages
Reproduction Steps
Add a child project to monorepo and observe
Possible Solution
No response
Additional Information/Context
No response
PDK version used
0.17.3/4
What languages are you seeing this issue on?
Typescript
Environment details (OS name and version, etc.)
Macos ARM

This is actually due to a change I made to projen to always create a .npmrc file for PNPM projects so that resolution-mode is set to highest. For reference see the projen issue here https://github.com/projen/projen/issues/2628 and where I upgraded the projen peer dependency version in PDK here: https://github.com/aws/aws-prototyping-sdk/pull/373
Nevertheless it is annoying having the extra .npmrc files, so I'll send a PR to remove any subproject .npmrc files if you're using the nx-monorepo


Improving Headings Detection (Related to Issue #30)

Issue #30

Added a new algorithm based on the elimination of the most common font style, basing on the hypothesis that the most common text style belongs to the body text.
Every other text style is then put in bins sorted by font size, which are then associated to different heading levels.
It seems to make improvements. Any critique is welcome!


Missing PNG files in the dataset (1-cell_image_library/dataset/train/train_x)

The directories  1-cell_image_library/dataset/train/train_x and 1-cell_image_library/dataset/train/train_x in the dataset are empty. It seems the code is expecting PNG files in these locations.
/home/lisergey/KartezioPaper/1-cil
:; python train_model.py
/usr/lib/python3/dist-packages/paramiko/transport.py:220: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[Kartezio - INFO] -  42 nodes registered.
[Kartezio - INFO] -  5 metrics registered.
[Kartezio - INFO] -  7 fitness registered.
[Kartezio - INFO] -  7 endpoints registered.
[Kartezio - INFO] -  5 stackers registered.
[<EMAIL_ADDRESS>global loadsave.cpp:241 findDecoder imread_('dataset/train/train_x/000_img.png'): can't open/read file: check file path/integrity```

Dear @slitvinov,
thank you for your feedback. The Cellpose dataset can be downloaded directly from the cellpose website of the authors.
I will update the README file to avoid future confusion.


pypi has package with same name

Hi @robinbaeyens, @Kiefersv:
someone stole our idea (before we even had it) and published a package on pypi with the same name as our lovely package.
We should consider using a different name instead, in order to be able to publish on pypi.
Here is a suggestion: gcm_utils
Cheers
Aaron

That's too bad... I guess we'll need to think of a new name then indeed.
Some more suggestions gcm_toolkit, py4gcm.
Or we can try to think of a good-old acronym :)

I vote for gcmtoolkit.
Please no acronym again :D :D :D

I like gcmtoolkit as well!
(But I disagree with Aaron, acronyms are amazing and should be as complicated as possible :D)

Lets dub it gcmtoolkit then. I will do the change as soon as #17 is merged!

I think as soon as we fixed (and secured) a name we should think about a logo. If anyone of you wants to have a shot at it that would be great, otherwise I would ask some more artistically talented friends of mine :D

Sure, go ahaid :D I'd say its more important to do some broadcasting of our software... Get other people to use it and stuff, but sure a logo is nice as well :D :D

Issue resolved, since we know changed the name to GCMT (short for GCMtoolkit) and reserved the pypi.org name.


Massive iRobot Create3 ROS2 issues!

I have 16 iRobot Create3's all running humble H.1.0 with RPi4's (running ubuntu 22.04 with ROS humble) communicating over USB-C. I setup all of these for a Lab and personally configured the USB connections. And multiple strange things are happening.
Originally they where running Galactic G.3.1 Not a single issue with anything. They all ran fine with goals and other commands. except for Dock goal, but that was ok and expected.
But I had them change to Humble H.1.0 because of the dock goal change/update because the PI's are running 22.04 which will only work with the humble version of ROS and the docking goal was changed in the versions. After the update there has been nothing but issues some of which could be fixed by "sudo apt upgrade" on the pi's but more issues would appear and then disappear. I would have them downgrade the firmware but issues would still arise.
The bot I have has had not one issue like they are having and my bot would run all day without any problems. Its a lab so we tell them what to put into their node.py within a ros2_ws/src and run the nodes.

Lets say you boot the bot. and it didn't have any problems running a goal or other commands directly from the CMD (SSH into PI) but you run the same commands from within a node. it works for the first few then no communication at all. every call will get hung up. if you reboot it will work again then will stop working.

Lets say you boot the bot and it didn't work with any commands at all and action servers are offline and you can't even see "ros2 param list" and no matter how many time you reboot the bot it changes nothing.

Lets say the next day after you were having the above 2 issues your bot would work perfectly fine and no issues would happen.

Lets say your bot was working fine one day and the next you were having issues 1 or 2.

Note all the bots boot with a solid red ring even mine which works fine. I have also personally looked over each groups code that was having problems and its all the same as mine which has no issue. I have cleaned out their src folder because a few groups ran "colcon build" in there which apparently can mess with ROS2 commands. This fixed some different issues they where having.
I have done extensive testing and trouble shooting to diagnose what on earth is happening and nothing is consistent either. This is maddening lol. I have been working on theses issues for 2 weeks solid. Not all robots are being problematic

The only things I have been able to determine is that because of how the issues are and a review of the logs it looks like a possible NTPD connection could be the problem? but I have tried to fix that by https://iroboteducation.github.io/create3_docs/setup/compute-ntp/ this guide. Now I have only done this once but it did not fix the robots issues.

Also it feels like after a while if sent goals from within a python node it will make the robot freak out and get hung up and stop excepting commands. So almost like just the python node send goals are not functioning correctly.

Or it feels like something is messed up when the build the package.

I can't even explain how much I have tried and done to get the bots to work correctly. Now some groups don't even have any issues, just like my bot. So if someone could please tell me I'm not going crazy lol. I could use the hand.

Hi @scottcandy34, your problem seems to be related to network issues. Are all the robots connected to the same Wi-Fi?
However, I want to point out an important point.
It looks like you updated the robots from Galactic to Humble only because of the Dock action.
There's a better solution! You could simply build the galactic branch of the irobot_create_msgs repo even if you are using Humble on your RaspberryPi!

They are all setup to be and at times they are all on the same connection. But when these issues were prominent only a few were on. All bots are on a different domain id as well. the PI's are on the same network. I can connect to the robots web page configs just fine for each bot, That's how I'm able to view the logs.
Network issues would make some sense but at the same time not because the bots have no issue being connected to the Wi-Fi. Only network type thing I can think of is NTP servers.
That is a good solution wish I realized that earlier, but that doesn't fix the overall problem that something is going on with these robots lol. The labs will end today so having them downgrade and do that is too much at this point. Plus once they go back to galactic some still would have issues, although at this point I don't think they would.
I would still like to figure out what on earth is going on so either you guys can put out a patch or I can finally know what not to do lol. My goal is at this point knowledge and preventative. Ill have the bots back here soon so I plan on working on them as is at least until I can figure out what is happening.

You are not alone @scottcandy34 I have 12 Turtlebot4s and intend to control them from a single computer. Unfortunately, I am in the same position. For now I am experimenting with 2 robots running Humble v1.2 and the create3 has always been the single point of failure; I have experienced numerous issues in Namespacing, NTP time sync issues and RMW preventing progress. To make matters worse the issues are really inconsistent leading to further confusion and frustration.

I can't even explain how much I have tried and done to get the bots to work correctly. Now some groups don't even have any issues, just like my bot. So if someone could please tell me I'm not going crazy lol. I could use the hand.
You are not crazy at all.

@Jayden-F Thank you for letting me know lol. It really is unfortunate. I plan to work on them more next week to get them all working since the students are done with them. If you get anywhere would you post and let me know? I'll do the same.

Hi all - I'm a professor who worked through the process of getting multiple creates with Pis online on the same network in the past year. I'd be happy to chat with either or both of you about how to do that. Feel free to email me at<EMAIL_ADDRESS>

@ipa-rar @Jayden-F I agree its become highly unpredictable.
I just went through all 16 of my bots and tested each one. I also upgraded to H.1.2 I copied over my code and replaced the students code also cleaned up all the build/install/log folders and ran it. everything worked fine without issues one PI needed to be updated 'sudo apt upgrade' which fixed its communication issue. The majority of the bots still boot with a RED ring before finishing boot. before this I had tested my pi on their bots and didn't have the issues they were having. now as I have said before I had inspected their code/PI before but couldn't find anything. but one thing I did change was setup.py file to mine which they could of done something that might of messed with the bots coms. I also didn't test when they were having the issues whether or not it was their code I could of removed the build/install/log and removed it from '.bashrc' source. but didn't think about that as a test until now. That would of proven if it was their code or the PI, based on all the other testing I had done.
Now this still doesn't explain all of the issues I have been having with their bots. I didn't test with more than 2 on the network but each bot is on its own domain so that shouldn't affect it. The bot will still occasionally have problems like it cant keep up with what you told it todo and acts slow or slow response.
In all my testing I have checked the bots log page in the web UI and very rarely have I seen it maxed out completely. now it does utilize 80 to 90 CPU percentage almost always. I didn't pay attention to the ram too much unfortunately. The bots for ROS are highly unpredictable.

What RMW_IMPLEMENTATION are you using? I know CycloneDDS will not work because in order for it to work, we would need to go into the create3 base and change the XML configuration file that the driver nodes are being launched under.
FastDDS has the option to add a Discovery Server in order to try and fix this, but it's quite a nightmare because the base disconnects from the WiFi when enabled (https://turtlebot.github.io/turtlebot4-user-manual/setup/networking.html) so the only way to run the server is to make the info travel through the usb0 IP (https://turtlebot.github.io/turtlebot4-user-manual/setup/networking.html)
I know this is not done correctly because even Turtlebot4 says multirobot setup is not currently supported, and this is the reason, it seems like unfinished work. Now it seems like they are adding beta features to tackle this: https://github.com/turtlebot/turtlebot4/issues/145
ROS 2 is simply too unreliable to work out of the box for communication between PCs in the same network. ROS 1 worked much better in this regard, and I think it's DDS's fault. If it worked as advertised, we wouldn't have to restart the daemon every three seconds.

We also ran into a lot of issues when using two robots (TurtleBot 4) on the same network. I tried to summarize the problem and potential workarounds in the following document: https://docs.google.com/document/d/18Uiy3k-HlsiR4ICyweVondgNwyK8dJw2Eda17PrzJdY/edit?usp=sharing Comments are appreciated.

Hi all - there are some new docs on a discovery server setup that I've been testing and seem to work quite well: https://iroboteducation.github.io/create3_docs/setup/discovery-server/
Note - I do not have the full Turtlebot4 setup, but I am using the Create 3 robot, a Raspberry Pi, RPLIDAR, and a computer running a linux VM. I'm hoping to test multiple robots in the next few days, but with one robot, this setup has been seamless.

@agrueneberg Thank you for your document, I am also experimenting with 2 TurtleBot4s (Hoping to move to 12) Yet I am roadblocked by the  exact same issues you have described below.

Our experience with multiple robots was poor (to say the least). At this time, it is not clear whether multiple robots can be used at the same time on the same network controlled by a single external computer. The main reason for this is that the Create 3 is hitting 100% CPU usage frequently and early, causing it to become unresponsive. This problem appears to occur even when using a single robot, but having more than one namespaced robot exasperates it.

I have also tried:
To add upon what you have. I also tried FastDDS with the discovery server to eliminate the multicast traffic. However, lifecycle nodes such as the nav2 stack hang and constantly wait for each other to be ready. Therefore, I am unable to use it at this time.
Yet to try:
I am currently experimenting with individual ROS2 networks for each robot using cyclone dds; Each Turtlebot4 will have its create3 disconnected from the network and will only communicate to the Rpi through a ROS2 network. The Rpi is then configured to use the usb ip network adapter as its preferred ROS2 network adapter isolating all ROS2 traffic between the Create3 and Rpi. We will use http Post and Get between the Rpi and centralised controller to send commands.
This is an insane work around, as we re-implementing what ROS2 and the RMW should be able to achieve, However, with the current state of the TurtleBot4 and RMW. It seem to be the only option. If anyone has better suggestion please let me know. I really need multiple TurtleBot4s working with a censtralised controller.

@Jayden-F Thanks for your feedback! I have added it to the document. We also had your idea, but had to drop it because other lab groups might want to use the robots, too. How would you configure Cyclone DDS to only communicate via usb0?

I'm currently finding some success by completely isolating the Create3 from anything except the RPi4. I do this in CycloneDDS to avoid the problem @Jayden-F described.
@agrueneberg  you can configure this in the new feature to override the RMW XML config, something like <NetworkInterfaceAddress>usb0</NetworkInterfaceAddress>
This is my config for the create3:
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
    <Domain id="any">
	<General>
            <NetworkInterfaceAddress>auto</NetworkInterfaceAddress>
	    <AllowMulticast>false</AllowMulticast>
            <MaxMessageSize>65500B</MaxMessageSize>
            <FragmentSize>4000B</FragmentSize>
        </General>
        <Discovery>
            <Peers>
                <Peer address="[RPI4 IP]"/>
            </Peers>
            <MaxAutoParticipantIndex>100</MaxAutoParticipantIndex>
            <ParticipantIndex>auto</ParticipantIndex>
        </Discovery>
        <Internal>
            <Watermarks>
                <WhcHigh>500kB</WhcHigh>
            </Watermarks>
        </Internal>
        <Tracing>
            <Verbosity>severe</Verbosity>
            <OutputFile>stdout</OutputFile>
        </Tracing>
    </Domain>
</CycloneDDS>

Multicast is disabled to reduce traffic and also increased packet size to decrease number of them processed. I only add the RPi4 as a Peer Address to ensure the create3 doesn't see anything else. Then I can worry about obtaining the topics I need from an external computer by "routing" them through the RPi4, which will discover all the other machines in my setup.

@agrueneberg I factory reset the create3; by default the create3 uses both network interfaces with cyclone dds. I then configure the Rpi to use the usb0. @royito55 method seems much better, and I am curious about the routing messages through the RPI4. Do you have more information on how this works?

Well it's like a workaround I though about, and it wouldn't be necessary for every setup. It's not related to DDS but the topics the create3 offers.
For example, if you make all your pi's see each other and a other PCs, then you can send nav2 goals to the pi's no problem.
But say you need to send dock commands to all your robots from an external PC, then you'd need to discover robot_n/dock coming from the create3, but with the setup I described, the PC doesn't see the create3. So then you need to write some ROS 2  nodes and run them in the pi's to address that, which I haven't lol.
For topics it will be easier by just running the great topic_tools relay package in the pi's.

We have just released G.5.2 and H.2.2, which reduce core robot loading by about 9%. Hopefully this, in combination with other configuration work that is in progress. will help. Please update your Create 3 robots to H.2.2, and let us know if you find a similar improvement.

@shamlian @alsora
Following these discussion on performance, I have tried H.2.3 with a custom Eclipse Cyclone DDS configuration on the robot as suggested above. A short while after a reboot with the DDS config script set for the robot, it crashed (permanent red LED), I could not power-off / restart it. Taking the battery out for less than 15 mins it kept immediately going back to a white spinning LED with no response when trying to hotspot it. Over 15 mins it doesn't turn-on when installing the battery and on the dock it goes back to a white spinning LED, with no change when going .
Is there a way to hardware reset it? I have tried holding the various buttons down for long periods (>20s) but still not getting a response (cannot access Wifi or hotspot the robot).
Thanks

@lukeopteran The one thing I can think to try is to connect to the robot over its usb0 interface once it is fully booted (wait about 6 or 7 minutes after it is powered on) and attempt a factory reset with something like curl -X POST http://<IP_ADDRESS>/api/factory-reset (this just triggers the same endpoint as if you had clicked the "factory reset" link from the webserver); if that doesn't work, please email<EMAIL_ADDRESS>so we can set up a call; I think that will be more efficient than trading messages here.

thanks will give it a try

@shamlian That worked, thanks very much, have switched to the Galactic FW!

Hi,
We have created a video explaining some of the issues the Create 3 has due to DDS networking, and provide a solution for it.
It is made for the TurtleBot 4, but it also applies to the Create 3 when connected to a computing unit like a raspberry pi.
I hope it helps!
https://www.youtube.com/watch?v=xmK2I0D5sas


Read git auth token from backend config

Hey, I just made a Pull Request!
While testing TechDocs from an internal GitLab repo, we found that the auth mechanism is not working properly and we were getting Error: remote authentication required but no callback set. Turned out that the token provided under integrations.gitlab wasn't being used and instead the GitlabPreparer was dependent on catalog.processors.gitlab.privateToken or the env var process.env.GITLAB_TOKEN. This PR adds support to check for the token first in the integrations config and then falls back to old behaviour. We have tested this with our internal GitLab.
Points to note

I am not sure of how GitHub or Azure auth logic is yet. If someone can verify it, that would be great.
It's my first major contribution here and first time working with TypeScript (or JavaScript). So please forgive my mistakes. Give me feedback and I will incorporate it :)

:heavy_check_mark: Checklist

[ ] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)

Not sure how we should do with the legacy token config. Is it better just not supporting it, or do we want a transition period?

This is great PR. I was struggling with the same issues while setting up GHE for TechDocs.

Let's keep the backwords compatibility for config now. cc @hooloovooo

Resolving them now. Curious to know if the community prefers a "squash and force push" or multiple commits approach in PRs.

I followed the "rebase, squash and force push" strategy, as I am used to it in my org. Let me know if the community prefers the multiple commits strategy.

Nice work @steelrooter ! Thanks for the pull request and welcome! :)


Got error when run demo

Hi,
I use
jetson nano 2 gb
Jetpack 4.6
CUDA 10.2
tensorrt 8.0
onnx 1.8
After compling, I run the demo, got error:
GPU has cuda devices: 1
----device id: 0 info----
GPU : NVIDIA Tegra X1
Capbility: 5.3
Global memory: 1979MB
Const memory: 64KB
SM in a block: 48KB
warp size: 32
threads in a block: 1024
block dim: (1024,1024,64)
grid dim:<PHONE_NUMBER>,65535,65535)
Building TRT engine.
[libprotobuf ERROR google/protobuf/text_format.cc:298] Error parsing text-format onnx2trt_onnx.ModelProto: 1:9: Message type “onnx2trt_onnx.ModelProto” has no field named “version”.
trt_infer: ModelImporter.cpp:682: Failed to parse ONNX model from file: …/…/model/pointpillar.onnx
: failed to parse onnx model file, please check the onnx version and trt support op!
So can you tell me how to solve this error or which version of onnx do you use?

@wuwenxuan0226 Please check the file model/pointpillar.onnx, and confirm that git-lfs downloaded the actual binary 19 MB file - if you open the file and see only human readable text, git-lfs didn't download it.

Hi,
Thanks for the reply. Now I can run it.
Many thanks.


FR : Open weekly note from Calendar view

A Feature Request to quickly open week notes :
When you press Enter on a day in the Calendar view, telekasten opens the corresponding daily note.
Why about having the same functionality to open the weekly note when selecting the week number ?

I think I'm going telekasten-adict :1st_place_medal:

Just see that it is already on the backlog :confused:
So a +1 vote for this functionality :+1:
By the way, in the calendar view, the last char of the line is a Space, so using $ goes after the week number :

(don't know if it comes from calendar.vim or from your fork : tell me if I better open an issue on the calendar.vim repo)

Just see that it is already on the backlog :confused:
So a +1 vote for this functionality :+1:
By the way, in the calendar view, the last char of the line is a Space, so using $ goes after the week number :

(don't know if it comes from calendar.vim or from your fork : tell me if I better open an issue on the calendar.vim repo)

I only changed some 'code' in calendar.vim to fix a bug with calendar action parameters. It also extends the vertical split by 1 column to avoid right margin cut-off if for some reason the signcolumn is on in the calender. It doesn't append spaces, though. So that must be in the original code and can probably be fixed. Something for the ever-growing backlog :smile:


Vulnerability roundup 63: zip-3.0: 1 advisory

search, files

[ ] CVE-2018-13410 (nixos-19.03)

Scanned versions: nixos-19.03: 5847485e3ec. May contain false positives.

The severity of this one seems very low.

Closing as Nixos 19.03 is no longer supported.


[11.x] Enhance database migrations

This PR does a lot of improvements and unlocks many possibilities on database migrations, also makes it easier to maintain!
New Features / Enhancements 🚀
Extend SQLite support to 3.26+
Laravel currently requires SQLite 3.35+, this PR extends SQLite support to 3.26+
Add and Drop Foreign Keys on SQLite
SQLite doesn't support adding / dropping foreign keys with alter table command, but as suggested on the docs, you may recreate the table with arbitrary changes, that's what this PR do to implement these commands.
This PR adds support for blueprint's $table->foreign(), $table->dropForeign() methods and ->constrained() modifier on SQLite:
Schema::table('posts', function (Blueprint $table) {
    $table->unsignedBigInteger('user_id');

    $table->foreign('user_id')->references('id')->on('users'); // Now works as expected on SQLite
});

Schema::table('posts', function (Blueprint $table) {
    $table->foreignId('user_id')
        ->constrained(); // Now works as expected on SQLite
});

Schema::table('posts', function (Blueprint $table) {
    $table->dropForeign(['user_id']); // Now works as expected on SQLite, no exception will be thrown anymore!
});

Fixes #51318, SQLite doesn't allow dropping a column if it is used in a foreign key constraint, this PR makes it possible:
Schema::table('sessions', function (Blueprint $table) {
    $table->dropForeign(['user_id']);
    $table->dropColumn('user_id');
});

// or with more magic...
Schema::table('sessions', function (Blueprint $table) {
    $table->dropConstrainedForeignId('user_id');
});

Add and Drop the primary key on SQLite
This PR adds support for blueprint's $table->primary(), $table->dropPrimary() methods, ->primary(), and ->primary(false)->change() modifiers when modifying table on SQLite:
Schema::table('posts', function (Blueprint $table) {
    $table->dropPrimary();              // Drop the current primary key
    $table->string('email');            // Add 'email' column
    $table->primary(['name', 'email']); // Add a composite primary key
});

Schema::table('posts', function (Blueprint $table) {
    $table->uuid('id')->primary();  // Add 'id' column and make it primary
});

Schema::table('posts', function (Blueprint $table) {
    $table->string('id')->primary(false)->change();  // Change the type of 'id' column to `string` and drop primary key
});

Preserve the order of commands
This is the most important change on this PR; When updating table and compiling blueprint commands to SQL, the order of commands wasn't respected:

First compile all modified columns at one query
Second compile all added columns at once
Then compile other commands

As discussed on #50925, this will causes issue, unexpected results and leads to error most of the time when updating table. For example, the following code raises The 'username' column already exists! SQL error on all databases, because add column command was compiled before rename column:
Schema::table('users', function (Blueprint $table) {
    $table->renameColumn('username', 'email');
    $table->string('username');
});

This PR fixes this by preserving the order of commands as declared on the blueprint. You don't have to call each command in a separate Schema::table anymore that breaks migration transaction!
Blueprint State for SQLite
As explained before, when updating table on SQLite, we have to recreate the table to do arbitrary changes. To do so we need to know the current state of the table on each command. This PR adds BlueprintState class to keep track of changes on the table. After this PR we are able to mix any combination of blueprint commands in a single Schema::table (as demonstrated on the added integration tests) and it works as expected in the same order!
Notes

Unlike other database drivers, foreign keys don't have name on SQLite, so obviously you can't drop a foreign key by its name on SQLite:
Schema::table('posts', function (Blueprint $table) {
    $table->dropForeign('posts_user_id_foreign'); // ❌ doesn't work on SQLite

    $table->dropForeign(['user_id']);             // ✔ works as expected
}

We probably need to remove withSqlite trait on orchestral/testbench package, it seems we don't need that anymore if it's merged?

Is there a reason, you mixed the SQLite and migration execution order changes into one PR instead of two separate ones?

I'm eager to see this merged!

@shaedrich we can't add SQLite features without fixing the migration commands' orders.

@hafezdivandari You could have created the SQLite PR as draft, added a comment concerning the PR merging order and have backmerged the main branch after the migration command order PR was merged

@shaedrich It's fine ;)

To be honest this just is too much for us to take on reviewing and maintaining at the moment and I'm not sure the return on investment is there for end users.

Hi @hafezdivandari. I think we should still give a part of your PR a chance but like Taylor said and like we said on the Passport PR keep in mind to keep PR's small and separate because reviewing and testing such a large PR as this is simply too much for us at this time. Sending in incremental PR's with the different topics from this PR will be much much more easier to review.
I feel like the most important part of this bit is the dropping of foreign keys in SQLite. Could we maybe just focus on that one for now so we can get that back into Laravel v11?
Besides that I went through your code changes and while I feel most of this looks okay I do have a question why you decided to move column modifications to different alter lines instead of packing them into a single query statement with add like it does now. Because with this PR it could lead to a head more query statements. Not sure if that's particularly bad but it did caught my eye and I'd like to get your opinion about it.

Hi @driesvints,

I think we should still give a part of your PR a chance but like Taylor said and like we said on the Passport PR keep in mind to keep PR's small and separate because reviewing and testing such a large PR as this is simply too much for us at this time. Sending in incremental PR's with the different topics from this PR will be much much more easier to review.

First of all, sorry for sending large PRs 😅 but this one is a little different from the PR on the Passport, separating this into more PRs (if technically possible) doesn't make it easier to review. Because although this PR fixes 2 main problems at the same time, it doesn't mean the changes can necessarily be sent on 2 separate PRs.
But I can add comments on each part of the code changes and explain in more details if you want. IMHO that would be really helpful when reviewing.

I feel like the most important part of this bit is the dropping of foreign keys in SQLite. Could we maybe just focus on that one for now so we can get that back into Laravel v11?

As I tried to explain on this comment earlier and this one, It is not possible to fix the drop foreign on SQLite (or any other commands that needs recreating the table on SQLite) without fixing the "order of commands" problem first.

Besides that I went through your code changes and while I feel most of this looks okay I do have a question why you decided to move column modifications to different alter lines instead of packing them into a single query statement with add like it does now. Because with this PR it could lead to a head more query statements. Not sure if that's particularly bad but it did caught my eye and I'd like to get your opinion about it.

Because preserving the order of commands matters. Please read the "Preserve the order of commands" section above.

We're gonna have another look at this but it could be a few weeks before we get to it. Thanks!

FWIW I’m running into the issue of not being able to drop foreign keys in SQLite and would very much appreciate having this fix in. It’s making a mess of things on my end.

So, I've reviewed this PR. I do have a few concerns. First, given my last 14 years of experience managing this framework, I feel like it's a 99.9% chance this PR will break existing applications given its size and nature of changes. Do we have anyone with large codebases that can try it?
Secondly, I fear for my ability to maintain it going forward. @hafezdivandari can you summarize the changes and how they work in plain English?

Thanks

Hi @hafezdivandari

We probably need to remove withSqlite trait on orchestral/testbench package, it seems we don't need that anymore if it's merged?

WithSqlite is not being used anywhere by default, and requires explicit invoke to trigger.

I started seeing some error with using Schema:::dropIfExists() with in-memory SQLite after the latest upgrade on Testbench.

1) Orchestra\Testbench\Tests\Databases\RefreshDatabaseUsingEventsTest::it_create_database_migrations
Error: Call to a member function getAttribute() on null

/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Database/Connection.php:1632
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Database/Schema/Grammars/SQLiteGrammar.php:39
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Database/Schema/Blueprint.php:293
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Database/Schema/Blueprint.php:209
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Database/Schema/Blueprint.php:130
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Database/Schema/Blueprint.php:116
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Database/Schema/Builder.php:564
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Database/Schema/Builder.php:446
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php:357
/Projects/orchestra/testbench-core/tests/Databases/RefreshDatabaseUsingEventsTest.php:36
/Projects/orchestra/testbench-core/src/Concerns/HandlesDatabases.php:47
/Projects/orchestra/testbench-core/src/Concerns/ApplicationTestingHooks.php:267
/Projects/orchestra/testbench-core/src/Concerns/ApplicationTestingHooks.php:110
/Projects/orchestra/testbench-core/src/Concerns/Testing.php:69
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Collections/helpers.php:236
/Projects/orchestra/testbench-core/src/functions.php:119
/Projects/orchestra/testbench-core/vendor/laravel/framework/src/Illuminate/Collections/helpers.php:236
/Projects/orchestra/testbench-core/src/Concerns/Testing.php:97
/Projects/orchestra/testbench-core/src/TestCase.php:65

https://github.com/orchestral/testbench-core/actions/runs/9866136206

@crynobone the error is on this line, it seems that connection's getPdo() is returning null which is wierd. I'm still investigating.

@crynobone I may have found the cause of issue:
The RefreshDatabase trait calls $connection->disconnect() so the connection's pdo is null on destroyDatabaseMigrations() method.
It seems that schema builder doesn't reconnect the connection!

Noted. Probably something Testbench specific. I will investigate further. Thanks @hafezdivandari

Adding a note here that this merge change appears to have affected some tests in my laravel-model-uuid package - specifically where we're making assertions against the shape of the resulting alter statements.
As I'm specifically testing that the generated queries are correctly formatted, I don't necessarily think this is a breaking change to end users as the resulting query is ultimately the same, just that now there are multiple queries rather than a single one.
Not sure this has tangible impact to framework consumers, though.

Thanks @michaeldyrynda, these changes were expected, as I explained on this comment before merge and the changes on framework's tests also show that.
Nice package by the way. You may also want to check PR #50355 for use in your code.

@hafezdivandari off course Taylor's feeling is right. It was a hard decision to make and the PR was open for 2 months.
Besides that, it was much easier for me to send this to 12.x. Since the patch release on Tuesday, I'm checking the issue board every hour and try to find related discussions on other packages too!

However, finding the root cause took quite some time, because this change wasn't appropriately documented (due to the size of the PR?) and it's breaking character wasn't obvious at first.

The PR description couldn't be more long I think, tried to cover every aspect + other comments on this thread. But I believe this maybe could be covered better on weekly Laravel newsletters after release to better inform the community.

I think this might be a breaking change. Disclaimer: the feature set is nice, so I'm not complaining, just thought to share the information.
One of our migrations started to fail on SQLite, beginning with Laravel 11.15:
https://github.com/vanilophp/framework/blob/4.1.0/src/Links/resources/database/migrations/2024_05_31_074502_add_root_item_to_link_group_items_table.php
Having Laravel 11.15 it fails:

Switching back to Laravel 11.14 solves the problem:

The linked example is from a library that supports Laravel 10.43+ and Laravel 11.0+
I believe we can find a workaround, like checking what Laravel version we have, and acting accordingly. But maybe you have a better suggestion.

Ohh, this is a large one... Breaking the tests for us.
Are there any more simple workarounds?

@ekandreas would you please share more info? Code? Exception?

Sorry, some kind of integrity violation in our seeder that caused an error.
We had to exclude some foreign keys in the schemas.
            if (env('DB_CONNECTION') == 'sqlite') {
                return;
            }


This plugin ought to suppress PossiblyUnusedClass/Method on common Laravel idoms

This is mainly a reference to this, but more broadly for the whole of Laravel: https://github.com/vimeo/psalm/issues/2489
List of issues to handle: https://github.com/vimeo/psalm/issues/2489#issuecomment-568410933
Currently getting 695 extra errors when I add --find-unused-code 😬 (on a different code base than above)
One of these days I'll put aside some time to fix this.

Currently, these are the errors that --find-unused-code are found:
Laravel
./tests/laravel-test.sh with --find-unused-code --no-cache
ERROR: UnusedClass - app/Console/Kernel.php:8:7 - Class App\Console\Kernel is never used (see https://psalm.dev/075)
class Kernel extends ConsoleKernel

ERROR: UnusedClass - app/Exceptions/Handler.php:8:7 - Class App\Exceptions\Handler is never used (see https://psalm.dev/075)
class Handler extends ExceptionHandler

ERROR: UnusedClass - app/Http/Controllers/Controller.php:10:7 - Class App\Http\Controllers\Controller is never used (see https://psalm.dev/075)
class Controller extends BaseController

ERROR: UnusedClass - app/Http/Kernel.php:7:7 - Class App\Http\Kernel is never used (see https://psalm.dev/075)
class Kernel extends HttpKernel

ERROR: PossiblyUnusedMethod - app/Http/Middleware/RedirectIfAuthenticated.php:20:21 - Cannot find any calls to method App\Http\Middleware\RedirectIfAuthenticated::handle (see https://psalm.dev/087)
    public function handle(Request $request, Closure $next, ...$guards)

ERROR: UnusedClass - app/Http/Middleware/TrustHosts.php:7:7 - Class App\Http\Middleware\TrustHosts is never used (see https://psalm.dev/075)
class TrustHosts extends Middleware

ERROR: UnusedClass - app/Providers/AppServiceProvider.php:7:7 - Class App\Providers\AppServiceProvider is never used (see https://psalm.dev/075)
class AppServiceProvider extends ServiceProvider

ERROR: UnusedClass - app/Providers/AuthServiceProvider.php:8:7 - Class App\Providers\AuthServiceProvider is never used (see https://psalm.dev/075)
class AuthServiceProvider extends ServiceProvider

ERROR: UnusedClass - app/Providers/BroadcastServiceProvider.php:8:7 - Class App\Providers\BroadcastServiceProvider is never used (see https://psalm.dev/075)
class BroadcastServiceProvider extends ServiceProvider

ERROR: UnusedClass - app/Providers/EventServiceProvider.php:10:7 - Class App\Providers\EventServiceProvider is never used (see https://psalm.dev/075)
class EventServiceProvider extends ServiceProvider

------------------------------
10 errors found
------------------------------
8 other issues found.
You can display them with --show-info=true
------------------------------
Psalm can automatically fix 1 of these issues.
Run Psalm again with
--alter --issues=PossiblyUnusedMethod --dry-run
to see what it can fix.
------------------------------

Checks took 9.01 seconds and used 413.896MB of memory
Psalm was able to infer types for 93.4426% of the codebase

Lumen
./tests/lumen-test.sh with --find-unused-code --no-cache
ERROR: UnusedClass - app/Console/Kernel.php:8:7 - Class App\Console\Kernel is never used (see https://psalm.dev/075)
class Kernel extends ConsoleKernel

ERROR: PossiblyUnusedMethod - app/Events/ExampleEvent.php:12:21 - Cannot find any calls to method App\Events\ExampleEvent::__construct (see https://psalm.dev/087)
    public function __construct()

ERROR: UnusedClass - app/Exceptions/Handler.php:12:7 - Class App\Exceptions\Handler is never used (see https://psalm.dev/075)
class Handler extends ExceptionHandler

ERROR: UnusedClass - app/Http/Controllers/ExampleController.php:5:7 - Class App\Http\Controllers\ExampleController is never used (see https://psalm.dev/075)
class ExampleController extends Controller

ERROR: UnusedClass - app/Http/Middleware/Authenticate.php:8:7 - Class App\Http\Middleware\Authenticate is never used (see https://psalm.dev/075)
class Authenticate

ERROR: UnusedClass - app/Http/Middleware/ExampleMiddleware.php:7:7 - Class App\Http\Middleware\ExampleMiddleware is never used (see https://psalm.dev/075)
class ExampleMiddleware

ERROR: UnusedClass - app/Jobs/ExampleJob.php:5:7 - Class App\Jobs\ExampleJob is never used (see https://psalm.dev/075)
class ExampleJob extends Job

ERROR: PossiblyUnusedMethod - app/Listeners/ExampleListener.php:16:21 - Cannot find any calls to method App\Listeners\ExampleListener::__construct (see https://psalm.dev/087)
    public function __construct()

ERROR: PossiblyUnusedMethod - app/Listeners/ExampleListener.php:27:21 - Cannot find any calls to method App\Listeners\ExampleListener::handle (see https://psalm.dev/087)
    public function handle(ExampleEvent $event)

ERROR: UnusedClass - app/Providers/AppServiceProvider.php:7:7 - Class App\Providers\AppServiceProvider is never used (see https://psalm.dev/075)
class AppServiceProvider extends ServiceProvider

ERROR: UnusedClass - app/Providers/AuthServiceProvider.php:9:7 - Class App\Providers\AuthServiceProvider is never used (see https://psalm.dev/075)
class AuthServiceProvider extends ServiceProvider

ERROR: UnusedClass - app/Providers/EventServiceProvider.php:7:7 - Class App\Providers\EventServiceProvider is never used (see https://psalm.dev/075)
class EventServiceProvider extends ServiceProvider

------------------------------
12 errors found
------------------------------
8 other issues found.
You can display them with --show-info=true
------------------------------
Psalm can automatically fix 3 of these issues.
Run Psalm again with
--alter --issues=PossiblyUnusedMethod --dry-run
to see what it can fix.
------------------------------

Checks took 5.94 seconds and used 207.604MB of memory
Psalm was able to infer types for 89.2857% of the codebase

IMO we can leave these errors in Lumen, because afaik those are actually unused in a fresh Lumen project:
ERROR: PossiblyUnusedMethod - app/Events/ExampleEvent.php:12:21 - Cannot find any calls to method App\Events\ExampleEvent::__construct (see https://psalm.dev/087)
    public function __construct()

ERROR: UnusedClass - app/Http/Controllers/ExampleController.php:5:7 - Class App\Http\Controllers\ExampleController is never used (see https://psalm.dev/075)
class ExampleController extends Controller

ERROR: UnusedClass - app/Http/Middleware/ExampleMiddleware.php:7:7 - Class App\Http\Middleware\ExampleMiddleware is never used (see https://psalm.dev/075)
class ExampleMiddleware

ERROR: UnusedClass - app/Jobs/ExampleJob.php:5:7 - Class App\Jobs\ExampleJob is never used (see https://psalm.dev/075)
class ExampleJob extends Job

ERROR: PossiblyUnusedMethod - app/Listeners/ExampleListener.php:16:21 - Cannot find any calls to method App\Listeners\ExampleListener::__construct (see https://psalm.dev/087)
    public function __construct()

ERROR: PossiblyUnusedMethod - app/Listeners/ExampleListener.php:27:21 - Cannot find any calls to method App\Listeners\ExampleListener::handle (see https://psalm.dev/087)
    public function handle(ExampleEvent $event)


[FEATURE_REQUEST] xsmall grid option

Is your feature request related to a problem? If so, please describe.
I see small grid value ends on ~600px on the phone ~380-450 only one column is shown
Describe the solution you'd like
can you add xsmall option to enable two or more columns option on small screens?
h(()=>["grid", >>>>>"grid-cols-1"<<<<<<<, sm:grid-cols-${e.grid.small}, md:grid-cols-${e.grid.medium}, lg:grid-cols-${e.grid.large}, xl:grid-cols-${e.grid.xlarge}, "gap-1", "lg:gap-2", "lg:gap-y-4"]);
return (n,a)=>{
Priority
Low (Nice-to-have)
Is this something you would be keen to implement
None

👋 @MG-Sky
Thanks for opening your first issue here!
To help make it easier for us to investigate your issue, please follow the contributing guidelines.
If you like this project, please ⭐ star our repo.