[BUG] when there are so many events , graphite does not handle it well

Describe the bug
We stored "events", "tags"  in mysql database , our system has about 1.5 million events and 10 million "tagging_taggeditem".  I usually got time out when querying the event data from timeout.
To Reproduce
This is what i see in mysql log:
SELECT events_event.id
FROM events_event, tagging_taggeditem
WHERE tagging_taggeditem.content_type_id = 12
AND tagging_taggeditem.tag_id IN (2,3,4,3344,23,239)
AND events_event.id = tagging_taggeditem.object_id
GROUP BY events_event.id
HAVING COUNT(events_event.id) = 6;
It takes about 2 minutes to complete, this will through the time out error.
Expected behavior
If we add an "AND when " in the query, it's much faster.
FROM events_event, tagging_taggeditem
WHERE tagging_taggeditem.content_type_id = 12
AND tagging_taggeditem.tag_id IN (2,3,4,3344,23,239)
AND events_event.id = tagging_taggeditem.object_id
AND events.event.when>'2020-01-01' and  AND events.event.when<'2020-02-01'
GROUP BY events_event.id
HAVING COUNT(events_event.id) = 6;
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
Graphite 1.1.5
Additional context
Add any other context about the problem here.

After debugging the code, the problem comes from this function Event.tagged.with_all(tags) , going further you will see it's from /opt/graphite/lib/python3.7/site-packages/tagging/models.py with the function get_intersection_by_model(...)  , in that function you will see that there is a join operation between event_events and tagging_taggeditem , so when there are millions record that join will become a real issue.
I'm not a Python expert, so this is my quick fix.
/opt/graphite/storage/models.py
def find_events(time_from=None, time_until=None, tags=None, set_operation=None):
    if tags is not None:
        if set_operation == 'union':
            query = Event.tagged.with_any(tags)
        elif set_operation == 'intersection':
            query = Event.tagged.with_intersection(tags)
        else:

#==============================================================================================
#My customize code
#==============================================================================================
if(len(tags)>2):
from tagging.utils import get_tag_list
tags_list = (get_tag_list(tags))
tags_id=[]
for idx in tags_list:
tags_id.append(idx.pk)
if len(tags_id)==0:
return []
additional_where =""
if time_from is not None:
additional_where = "%s  and  when >= '%s' " % (additional_where, time_from)
if time_until is not None:
additional_where = "%s   and when <= '%s' " % (additional_where,time_until)
sql = """
SELECT events_event.id
FROM events_event, tagging_taggeditem
WHERE tagging_taggeditem.content_type_id = 12
AND tagging_taggeditem.tag_id IN (%s)
AND events_event.id = tagging_taggeditem.object_id
%s
GROUP BY events_event.id
HAVING COUNT(events_event.id) = %s;""" % ( ",".join(map(str,tags_id)),additional_where,len(tags_id))
event_query  = Event.objects.raw(sql)
event_ids=[]
for event_id in event_query:
event_ids.append(event_id.id)
                sql = """
SELECT * FROM events_event
WHERE (events_event.id IN (%s) )
"""  % ",".join(map(str,event_ids))
                return list(Event.objects.raw(sql))
            else:

#==============================================================================================
query = Event.tagged.with_all(tags)
else:
query = Event.objects.all()
    if time_from is not None:
        query = query.filter(when__gte=time_from)

    if time_until is not None:
        query = query.filter(when__lte=time_until)

    result = list(query.order_by("when"))
    return result

Hi @mrtypn,
Thanks for your report! Yes, storing tags in database is sub-optimal, that's known issue - Redis works much faster. I'm not sure that your solution is generic enough to include it in the code.
I'll include it below for clarity. Thoughts, @DanCech @piotr1212 @ploxiln @cbowman0 ?
 def find_events(time_from=None, time_until=None, tags=None, set_operation=None):

        if tags is not None:
            if set_operation == 'union':
                query = Event.tagged.with_any(tags)
            elif set_operation == 'intersection':
                query = Event.tagged.with_intersection(tags)
            else:
#==============================================================================================
#My customize code
#==============================================================================================
                if(len(tags)>2):
                    from tagging.utils import get_tag_list
                    tags_list = (get_tag_list(tags))
                    tags_id=[]
                    for idx in tags_list:
                          tags_id.append(idx.pk)
                    if len(tags_id)==0:
                        return []
                    additional_where =""
                    if time_from is not None:
                        additional_where = "%s  and  `when` >= '%s' " % (additional_where, time_from)
                    if time_until is not None:
                        additional_where = "%s   and `when` <= '%s' " % (additional_where,time_until)
                    sql = """
    SELECT events_event.id
    FROM events_event, tagging_taggeditem
    WHERE tagging_taggeditem.content_type_id = 12
    AND tagging_taggeditem.tag_id IN (%s)
    AND events_event.id = tagging_taggeditem.object_id
    %s
    GROUP BY events_event.id
    HAVING COUNT(events_event.id) = %s;""" % ( ",".join(map(str,tags_id)),additional_where,len(tags_id))
                    event_query  = Event.objects.raw(sql)
                    event_ids=[]
                    for event_id in event_query:
                           event_ids.append(event_id.id)

                    sql = """
    SELECT * FROM events_event
    WHERE (events_event.id IN (%s) )
    """  % ",".join(map(str,event_ids))
                    return list(Event.objects.raw(sql))
                else:
#==============================================================================================
                    query = Event.tagged.with_all(tags)
        else:
            query = Event.objects.all()

        if time_from is not None:
            query = query.filter(when__gte=time_from)

        if time_until is not None:
            query = query.filter(when__lte=time_until)

        result = list(query.order_by("when"))
        return result

@deniszh this is talking about events, not series tagging.
Testing this locally I definitely see the WHERE clauses for events_event.when being added to the query.  Are you maybe running an old version of django or django tagging?
I do note that there is no index on the when column, it may be more efficient if we add an index on that column so that the database can filter more efficiently.
To have django print all queries for debugging you can add this to your local_settings.py:
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
        }
    },
    'loggers': {
        'django.db.backends': {
            'handlers': ['console'],
            'level': 'DEBUG',
        },
    }
}

@DanCech  ,
The issue  is from this function query = Event.tagged.with_all(tags) in this function
def find_events(time_from=None, time_until=None, tags=None, set_operation=None):
That function does not use the date , it's in tagging package

Riiight, sorry, I didn't check patch thoroughly and didn't understand it relates to different type of tags. 🤦
Not sure how to properly propagate date info to  Event.tagged.with_all(tags) though....

@mrtypn that function executes a query against tagging_tag to get the tag ids, but it generates a QuerySet object for the main query, which subsequently has the filter added to it and then gets executed when it's cast to a list.  With query logging enabled I get queries like this:
(0.000) SELECT `tagging_tag`.`id`, `tagging_tag`.`name` FROM `tagging_tag` WHERE `tagging_tag`.`name` IN ('blah') ORDER BY `tagging_tag`.`name` ASC; args=(u'blah',)

and
(0.001) SELECT `events_event`.`id`, `events_event`.`when`, `events_event`.`what`, `events_event`.`data`, `events_event`.`tags` FROM `events_event` , `tagging_taggeditem` WHERE ((`tagging_taggeditem`.content_type_id = 8) AND (`tagging_taggeditem`.tag_id = 1) AND (`events_event`.`id` = `tagging_taggeditem`.object_id) AND `events_event`.`when` >= '1970-01-01 00:00:00' AND `events_event`.`when` <= '2020-05-04 13:49:13') ORDER BY `events_event`.`when` ASC; args=(8L, 1L, u'1970-01-01 00:00:00', u'2020-05-04 13:49:13')

It's possible that there's another path inside tagging that executes another intermediate query depending on the results of the first query, but I haven't gone digging.  If you could enable sql logging as above and share the results you get that might help understand what's going on.


CoAP over TCP/TLS

RFC 8323 specifies CoAP over TCP instead of UDP. Roughly speaking, the protocol is the same as in the UDP case, but measures that let the application detect duplicate messages and resolve ordering issues are not included within the TCP version of CoAP.
Questions that come to my mind are:

Is there any intention/timeline to support CoAP over TCP (or even TLS)?
Does an integration of the TCP case make sense to include in the current library? Can one estimate the refactoring efforts?

Thanks!

@gbrgr I think concrete plans or a timeline does not exist :-) So in fact check the code and we are happy to get a PR :-)
In fact when I read the code (veeeery rough checkup) correctly then it already could work if you initialize your tcp server outside and pass it as first parameter to the "listen" method - it needs to be an EventEmitter and emit events like the dgram socket. In fact mainly two places interact with dgram ... the server class and the agent class, so refactoring also to officially support tcp should be not too complex


Bug: document scroll is active in Safari

title

For bug reports, please include:

a jsfiddle in order for us to be able to reproduce the issue.
expected behavior of the plugin
actual behavior

JSFiddle template: https://jsfiddle.net/limon/ad3quksn/

See Contributing.md


"deis doctor" interface for on-demand debug submission

Currently, workflow-manager is capable of submitting a deis component+version payload to versions.deis.com according to a regular interval. Leverage this existing wiring, as appropriate, to enable on-demand submissions of deis component cluster status data for the purpose of engaging w/ official support channels. An HTTP interface is an appropriate implementation.

fixed in #70


Unit Tests Coverage.

This PR is a WIP for all unit testing on this library.

Codecov Report

Merging #6 into develop will increase coverage by 12.51%.
The diff coverage is 100%.

@@              Coverage Diff               @@
##             develop       #6       +/-   ##
==============================================
+ Coverage      15.38%   27.89%   +12.51%
- Complexity       215      217        +2
==============================================
  Files             21       21
  Lines            468      466        -2
==============================================
+ Hits              72      130       +58
+ Misses           396      336       -60

Impacted Files
Coverage Δ
Complexity Δ

src/Auth/Manager.php
100% <ø> (+100%)
4 <0> (-1)
:arrow_down:

src/Operations/Common/HasName.php
100% <100%> (+100%)
3 <2> (+1)
:arrow_up:

src/Operations/Common/HasParameters.php
100% <100%> (+100%)
9 <0> (ø)
:arrow_down:

src/Operations/Common/SerializesJson.php
100% <100%> (+100%)
13 <0> (+2)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6ff097...853d892. Read the comment docs.


Docs: remove [sudo] from Installation, add pre-reqs and packaging tools reference

Cookiecutter version: master
Template project url: n/a
Python version: n/a
Operating System: n/a

Description:
In the installation instructions, there are an optional [sudo]s, which implies installing cookiecutter into a system Python instead of a virtual environment.
In Pyramid, we do more user hand-holding, starting from installing Python, then satisfying Requirements for Installing Packages, and finally creating a virtual environment into which we install Pyramid for either UNIX or Windows.
Would cookiecutter maintainers be interested in an approach similar to Pyramid's? I would submit the PR, if amenable.
What I've run:
n/a

Hey @stevepiercy! 👋
I think that's a great idea. Would love to review a PR.

FWIW pip install --user cookiecutter and a note about ensuring that the relevant bin folder $HOME/.local/bin (not sure what it is on windows) is on your path is much closer to a best practice these days than anything else involving pip installing a system-wide tool.

From pip's help:

--user                      Install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%\Python on Windows. (See the Python documentation for site.USER_BASE for full details.)

@hackebrot I have some docs to finish up in Pyramid first, then I'll return to this issue.

@hackebrot see #880

Closed by merging #880


Translate reference/kubectl/overview.md into Korean

Closes #21551
This PR contains the following translated document:

content/ko/docs/reference/kubectl/overview.md
content/ko/docs/reference/kubectl/_index.md

And also fixes the reference links to the above translated document for the following files:

content/ko/docs/tasks/tools/install-kubectl.md
content/ko/docs/setup/production-environment/windows/user-guide-windows-containers.md
content/ko/docs/concepts/overview/kubernetes-api.md
content/ko/docs/concepts/cluster-administration/manage-deployment.md
content/ko/docs/reference/kubectl/cheatsheet.md
content/ko/docs/reference/_index.md
content/ko/docs/reference/using-api/api-overview.md

/assign @gochist

Deploy preview: https://deploy-preview-21552--k8s-dev-ko.netlify.app/ko/docs/reference/kubectl/overview/

@pjhwa 감사합니다!
/lgtm

/approve


Update Age calculatortkinter.py

What type of PR is this? (check all applicable)

[ ] 🚀 Added Name
[ ] ✨ Feature
[ ] 🌟 stared the repo
[ ] 🐛 Grammatical Error
[ ] 📝 Documentation Update
[ ] 🚩 Other

Description
Add Link of GitHub Profile

Please merge the code.


correct scaling approximation and ensure raster has sufficient resolution

Given the following SVG:
<svg height="100" width="100">
  <circle cx="50" cy="50" r="40" stroke="black" stroke-width="3" fill="red" />
</svg>

When placed in a container that is double that size with BoxFit.fill
Before:

After:

This also ensures that DPR is properly used to scale the picture


Update misleading statement.

Following statement in this section is misleading.

Note the following important changes to make before you upgrade to any of the available minor versions:

It should read.

Note the following important changes before you upgrade to any of the available minor versions:

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: ec484163-5d73-af3a-3d7b-9e0d421d2824
Version Independent ID: 42da7aed-a33d-e331-fb7f-4fb4b40e5813
Content: Supported Kubernetes versions in Azure Kubernetes Service (AKS). - Azure Kubernetes Service
Content Source: articles/aks/supported-kubernetes-versions.md
Service: azure-kubernetes-service
GitHub Login: @Nickomang
Microsoft Alias: nickoman

@vermacodes
Thanks for your feedback! We will investigate and update as appropriate.

@vermacodes
Thanks for reporting this! PR has created for this issue and the changes should go live soon.


grad transformed functions should not return NumPy arrays

This behavior looks really strange to me:
In [1]: import jax

In [2]: jax.grad(lambda x: x)(1.0)
Out[2]: array(1.)

In [3]: type(Out[2])
Out[3]: numpy.ndarray

I would expect JAX to always return DeviceArray objects or builtin Python scalars.

It looks like this has been fixed somewhere along the way:
In [1]: import jax

In [2]: jax.grad(lambda x: x)(1.0)
Out[2]: DeviceArray(1., dtype=float32, weak_type=True)


chore: Upgrade argocd

What type of PR is this?

Uncomment only one  /kind line, and delete the rest.
For example, > /kind bug would simply become: /kind bug

/kind bug
/kind cleanup
/kind failing-test
/kind enhancement
/kind documentation
/kind code-refactoring

What does this PR do / why we need it:
Have you updated the necessary documentation?

[ ] Documentation update is required by this PR.
[ ] Documentation has been updated.

Which issue(s) this PR fixes:
Fixes #?
Test acceptance criteria:

[ ] Unit Test
[ ] E2E Test

How to test changes / Special notes to the reviewer:

/retest

Test failure
    logger.go:42: 14:22:13 | 1-031_validate_toolchain/1-check | Kustomize version mismatch. Should be v5.4.2, is v5.4.3
    logger.go:42: 14:22:13 | 1-031_validate_toolchain/1-check | Helm version mismatch. Should be v3.15.2, is v3.15.4
    logger.go:42: 14:22:13 | 1-031_validate_toolchain/1-check |

lgtm

/cherry-pick v1.15


ML - displaying all Python and R packages (including system ones)

In manage database packages, we should show all python and R packages, even the system ones. However
, user should not be able to uninstall the system packages. The UI doesn't support showing that to user but the service prevent user from uninstalling system packages.
Fixes #10487

Coverage increased (+0.1%) to 34.373% when pulling 5b6a5cf16760a1d5781a726fa651a2bd2987267c on llali/ml_readonly_packages into 107b3171f29416a92787689d3df8b9b3fd79725b on master.


Rancher can't perform App upgrade from helm2 to helm3

Use case:
I have a helm2 chart(apiversion: 1) named 'myChart' imported as HELM2 Catalog  and an App ('myApp') is deployed from it.
I want to increase the apiversion of the chart to 2.
According to do documentation this will require the remove/readd of the catalog as helm3
https://rancher.com/docs/rancher/v2.x/en/helm-charts/legacy-catalogs/#catalog-helm-deployment-versions
When you create a custom catalog, you will have to configure the catalog to use either Helm 2 or Helm 3. This version cannot be changed later. If the catalog is added with the wrong Helm version, it will need to be deleted and re-added.
Unfortunately it doesn't work,Rancher shows the catalog as a Helm3 one, but the App upgrade fails :
 Error: apiVersion 'v2' is not valid. The value must be "v1" : exit status 1
The deployed App's view in API section shows :   "helmVersion": "rancher-helm".
So as I tried the other way around :
Starting with a clean install:
Using the same helm2 chart but importing the Catalog as Helm3, the App is deployed.
Then I remove the catalog and readd it as Helm2.  Now the upgrade goes well, but
The deployed App's view in API section shows :   "helmVersion": "helm_v3".
So as turned out even if I change the catalog's helm version it will not change the way how it manages already deployed releases.
The documentation doesn't say anything about it. It only states that the catalog should be removed/readd.
So my question is : How to upgrade an already deployed APP from helm2 to helm3?
note: The only change in the chart is the Chart.yaml's apiversion field.
Rancher version: 2.4.8

Seems this related https://github.com/rancher/rancher/issues/26407
In my case helm3 v1 -> helm3 v2 fails but with the same error and seems because rancher uses helm2 binary instead of helm3

I have the same problem. I added the Helm catalog long time ago. At this time Helm 3 wasn't out.
Now a chart has transitioned from Helm 2 to Helm 3 (Rook Operator). I deleted the v2 Helm catalog and readded it as v3 Helm catalog. The upgrade of the app still fails with the error message @hatharom has mentioned.
Is there a way to manually upgrade an app from v2 to v3 without deleting and recreating it? (in my case: Deleting the Rook Operator Helm chart would delete the distributed Ceph storage. Rebuilding everything from backup would be a really huge effort.)

Our setup is relatively new so I was able to backup various things/dbs and redeploy helm 3 chart apps.
I looked into migrating the apps from helm v2 to v3, but I eventually gave up and did everything manually.
There was this issues opened https://github.com/rancher/rancher/issues/28198, but nothing happened.
Rancher docs now contain https://rancher.com/docs/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2/, but this really only talks about if you installed the Rancher management cluster with helm 2. Same for this doc https://rancher.com/docs/rancher/v2.x/en/installation/resources/helm-version/
Both link to this https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/, this does not look like it is useful for this. No matter where I setup the plugin, it could never see the apps installed to migrate them. I did this on the hosts as well as different rancher system workload pods.
Hope this helps a little and that you can find something I could not.

Hi all,
I get the same issue with rook operator create in v1.4.6.
I have recreated the catalog rook with helm_v3,
But get  error during upgrade of the rook operator in v1.5.5: Error: apiVersion 'v2' is not valid. The value must be "v1" : exit status,
I don't want to recreate the app, because it means to loose all the ceph cluster.
Is there a way to force with rancher API the "helmVersion": "helm_v3" on the existing rook operator app ?

Offtopic, because it has nothing to do with the "bug":
@konvergence I have tried everything and found no way to manually upgrade the deployment without recreating the app. I ended up to do a disaster recovery of the Rook Ceph Cluster. I used the guide provided at https://github.com/rook/rook/blob/master/Documentation/ceph-disaster-recovery.md#adopt-an-existing-rook-ceph-cluster-into-a-new-kubernetes-cluster . Have a working (!!!) backup of everything (each Kubernetes node and the Rancher instance), delete the app, add the helm catalog as v3, and then follow the guide step by step. Last step is to increase the monitor count to your previous setting (this step is missing in the guide).

/reopen


Exceptions when removing or renaming an element that is open in the editor and you save the editor.

I am not really sure what the solution to this is. One approach would be to disable the element's menu items when it is open for edit. When we turn everything into a window this might not be as big of an issue depending on what we allow to be modal vs non-modal. This needs more investigation. It is possible that i handle this with some of the other elements. I know there was a problem with the IAS element.

Do we think that this is critical and we handle this now? I don’t have the schedule in front of me now, but I think that the window-handling task is toward the end of the period of performance.
Or, is there a short-term fix you could do now and we re-evaluate when we get to the window task?

On Nov 2, 2021, at 6:03 AM, codymccoy @.***> wrote:

Assigned #174 to @rnugent3.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Thank you.
Best,
R Nugent III

On Nov 2, 2021, at 8:18 AM, codymccoy @.***> wrote:

It is definitely something that needs to be addressed at some point. At some point this week I will try to get a better grasp on the issue and come up with some suggested solutions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

I do not see the exception being thrown. Really all we need now is better validation on this weird behavior, but this is not critical.

@codymccoy I think it would be reasonable to disable the element's menu items when it is open for edit. What is the level of effort to do that?

I just tested this and I don't get any exceptions. I think the original problem was occuring because we used to use the element's name to handle the database stuff. Now we are using an ID. If you rename the element while the editor is open and then click save, it just updates as you would expect. If you remove the element while the editor is open and then click save on the editor, nothing happens because it tries to find the element to update in the DB and it can't find it.
@rnugent3 I think we can close this one.

This should be fixed now. I wrote a method in SavingBase called IDExistsInDB(). When "saving existing" if the id doesn't still exist then i switch it to do a "save new". This should be in main by the end of the day.

Cool! Thank you.

This works as expected


Wake up with http request

hi
sorry but i need help
what is the complete string to let the tablet wake up from screensaver?
http://<IP_ADDRESS>:2971.................... ????
and for  reload the start url  or restart the app
thanks
tom

Check the documentation under HTTP Commands https://github.com/thanksmister/wallpanel-android#commands.


Feat: Add Option to check CMS against a given X509 Certificate

The new introduced -c option for all validating-methods (validate-string, validate-cert, validate-file) adds the possibility to check whether the checked CMS was signed by the X509Certificate provided with -c option.
Result of the check will be printed out:
Matches Given Certificate: yes

or
Matches Given Certificate: no

This PR also adds more information about the signer certificate to the output: Certificate SHA-256 Thumbprint, Certificate Issuer Subject.

closes #29


refactor: Weaver Build and Workflows

Refactoring weaver directory so that the build works:

All go modules name corrected from hyperledger-labs/weaver-dlt-interoperability to hyperledger/cacti.
All hyperlinks to github.com/hyperledger-labs/weaver-dlt-interoperability corrected to github.com/hyperledger/cacti/weaver
Enabled Weaver Testing workflows, with refactoring so that it works in cacti.
Added samples files from weaver, missed during merge.
Fix NodeSDK GetSubscriptionState Bug (update from hyperledger-labs/weaver-dlt-interopability repo)

Chore:

Update my name (sanvenDev -> sandeepnRES)
Added corda generated files, and corda serializationwhitelist code in .dcilintignore.
Delete all Jenkinsfile that are not required anymore.

Closing this and opening new #2297 to resolve conflicts.


fix!: dirty hack to signing successfully the first data commitment range

Overview
This is because we included the  latest core release in the app v0.13.2 without including the corresponding change in the state machine.
For BSR, this case will never be hit since we will stop at the last unbonding height. But, I think it's better if we have the E2E passing to be more sure this is working.

Checklist

[ ] New and updated code has appropriate documentation
[ ] New and updated code has new and/or updated testing
[ ] Required CI checks are passing
[ ] Visual proof for any user facing features like CLI or documentation updates
[ ] Linked issues closed with keywords

This will  be only for this release... won't be merged to main. So I guess no need for an issue. What do you think? @evan-forbes

Codecov Report

:exclamation: No coverage uploaded for pull request base (v0.2.0-app-v0.13.2-beta@765ef7d). Click here to learn what that means.
The diff coverage is n/a.

@@                    Coverage Diff                     @@
##             v0.2.0-app-v0.13.2-beta     #340   +/-   ##
==========================================================
  Coverage                           ?   43.57%
==========================================================
  Files                              ?       26
  Lines                              ?     1999
  Branches                           ?        0
==========================================================
  Hits                               ?      871
  Misses                             ?     1005
  Partials                           ?      123


handle search results pagination with a generator

Original report by me.

In python API, use a generator / yield to automate the query of paginated search results

Make download_all() method auto increment over pages to download all found pages at once.

Add a parameter to download_all() to enable pages auto-iteration or not
On search, save search params in SearchResults object the same way it is done at EOProduct level
we may add a generator to SearchResults for auto-iteration, like a .next_page() method

When using download_all on a paginated SearchResult, page iterator should be automatically used to fetch next pages products


Merge SystemPool and support/Pool

Problem
We have too many pool implementation.
Change overview
There are 3 changes:
Add a PostLambda to post a generic lambda into system event queue #9678
Add a PostLambda to post a generic lambda into system event queue. This will decouple system object from network endpoints (TCPEndpoint, UDPEndpoint).
There is several limitation of the lambda

The lambda must be trivially copyable. Because the lambda is memcpy into the queue. It means that the lambda can only capture pod data (Including pointers).
The size of the lambda must less than the queue element size. The size of the lambda is the sum of all captured fields. The body of the lambda is encoded in the type info of the lambda, which is captured by a static function LambdaProxy::Call, and resolved into a function pointer at compile time, assigned to LambdaBridge::LambdaProxy into the queue element.

Unify SystemPool and lib/support/Pool #9590
Unify the interface of system/SystemPool and lib/support/pool. This PR change the interface of SystemPool to match lib/support/Pool. The pool provides following interface functions:
    T * CreateObject(Args&&... args)
    void ReleaseObject(T * object)
    template <typename Function> bool ForEachActiveObject(Function && function)

And also provides statistics interface:
    void ResetStatistics() {}
    void GetStatistics(Stats::count_t & aNumInUse, Stats::count_t & aHighWatermark) {}

Following features are decoupled from the pool implementation:

Reference counter: if the target require this feature, it can derive from lib/core/ReferenceCounted class
DeferredRelease: this feature should be orthogonal to pool implementation, the target class should implement it by itself.

Files changes:
Removed

src/system/SystemObject.cpp
src/system/SystemObject.h

Implementation on platforms with heap (POSIX):

src/system/SystemPoolHeap.h

Implementation on platforms w/o heap (Embedded):

src/system/SystemPoolNonHeap.h
src/system/SystemPoolNonHeapBitmapAllocator.cpp
src/system/SystemPoolNonHeapBitmapAllocator.h

Statistics:

src/system/SystemPoolStatistics.h

Select implementation based on pre-defined macro

src/system/SystemPool.h

Merge SystemPool and support/Pool #9748 (this PR)
Remove lib/support/Pool, migrate all its usage to system/SystemPool
Testing
Verified using unit tests

This PR contains all the changes of https://github.com/project-chip/connectedhomeip/pull/9590, should we close #9590 and focus on this one instead?

@kpschoedel could you help review this given you've been doing a fair bit of work in the System area?

@kpschoedel could you help review this given you've been doing a fair bit of work in the System area?

Kevin is OOO until end of the month.

Problem
We have too many pool implementation.
Change overview
Merge SystemPool and support/Pool

Can you please explain your changes?

Testing
Verified using unit tests

@yufengwangca

This PR contains all the changes of #9590, should we close #9590 and focus on this one instead?

@mspang

Can you please explain your changes?

Due to limitation of github, I can let this PR depends on other PRs, so I included all contents of related PRs into this one. We can either merge them one by one, or merge this PR and close others.

Please don't use recursive mutexes.

@mspang ,
The recursive mutex comes about with the original object pool design which was more than an allocator: it also allowed the sdk to iterate over all allocated objects in the pool.  So instead of properly tracking state, we shoe-horned this iteration facility into an allocator to allow the sdk to do a foreach on all objects of a given class without actually having to actively track anything.
When we moved to the heap implementation, this became a very difficult problem.  And now to do an object release or mutation in the middle of the foreach loop, you need the recursive mutex.
Is there a way that we can remove the need to iterate over all pool objects in the first place?  This would make things much, much simpler.

Please don't use recursive mutexes.

@mspang ,
The recursive mutex comes about with the original object pool design which was more than an allocator: it also allowed the sdk to iterate over all allocated objects in the pool. So instead of properly tracking state, we shoe-horned this iteration facility into an allocator to allow the sdk to do a foreach on all objects of a given class without actually having to actively track anything.
When we moved to the heap implementation, this became a very difficult problem. And now to do an object release or mutation in the middle of the foreach loop, you need the recursive mutex.
Is there a way that we can remove the need to iterate over all pool objects in the first place? This would make things much, much simpler.

I would agree with that statement.
This locks the set (why a set?), then makes a copy of it, but keeps the original lock while iterating the copy. Why did we make the copy? Because a reentrant mutex doesn't make reentrant modification safe. Why hold the lock while working with a copy? There's no logic to this.

This locks the set (why a set?), then makes a copy of it, but keeps the original lock while iterating the copy. Why did we make the copy? Because a reentrant mutex doesn't make reentrant modification safe. Why hold the lock while working with a copy? There's no logic to this.

The iterator of std::set is not stable, if the handler deleted the object (which happens regularly when handling events), it wil crash in the middle of the loop, so I need to create a copy of the set.
I'll change it such that it only lock when creating the copy, and use normal mutex instead of a recursive mutex.

This locks the set (why a set?), then makes a copy of it, but keeps the original lock while iterating the copy. Why did we make the copy? Because a reentrant mutex doesn't make reentrant modification safe. Why hold the lock while working with a copy? There's no logic to this.

The iterator of std::set is not stable, if the handler deleted the object (which happens regularly when handling events), it wil crash in the middle of the loop, so I need to create a copy of the set.
I'll change it such that it only lock when creating the copy, and use normal mutex instead of a recursive mutex.

The copy becomes invalid as soon as the mutex is released.

This locks the set (why a set?), then makes a copy of it, but keeps the original lock while iterating the copy. Why did we make the copy? Because a reentrant mutex doesn't make reentrant modification safe. Why hold the lock while working with a copy? There's no logic to this.

The iterator of std::set is not stable, if the handler deleted the object (which happens regularly when handling events), it wil crash in the middle of the loop, so I need to create a copy of the set.
I'll change it such that it only lock when creating the copy, and use normal mutex instead of a recursive mutex.

The copy becomes invalid as soon as the mutex is released.

This locks the set (why a set?), then makes a copy of it, but keeps the original lock while iterating the copy. Why did we make the copy? Because a reentrant mutex doesn't make reentrant modification safe. Why hold the lock while working with a copy? There's no logic to this.

The iterator of std::set is not stable, if the handler deleted the object (which happens regularly when handling events), it wil crash in the middle of the loop, so I need to create a copy of the set.
I'll change it such that it only lock when creating the copy, and use normal mutex instead of a recursive mutex.

I'm not sure it's worth writing that up until we have better design consensus.

Please don't use recursive mutexes.

@mspang ,
The recursive mutex comes about with the original object pool design which was more than an allocator: it also allowed the sdk to iterate over all allocated objects in the pool. So instead of properly tracking state, we shoe-horned this iteration facility into an allocator to allow the sdk to do a foreach on all objects of a given class without actually having to actively track anything.
When we moved to the heap implementation, this became a very difficult problem. And now to do an object release or mutation in the middle of the foreach loop, you need the recursive mutex.
Is there a way that we can remove the need to iterate over all pool objects in the first place? This would make things much, much simpler.

I would agree with that statement.
This locks the set (why a set?), then makes a copy of it, but keeps the original lock while iterating the copy. Why did we make the copy? Because a reentrant mutex doesn't make reentrant modification safe. Why hold the lock while working with a copy? There's no logic to this.

No, the current implementation use std::mutex, it is not the original System Pool design, we should change back to std::mutex

@yufengwangca

This PR contains all the changes of #9590, should we close #9590 and focus on this one instead?

@mspang

Can you please explain your changes?

Due to limitation of github, I can let this PR depend on other PRs, so I have to include all contents of related PRs into this one. We can either merge them one by one, or merge this PR and close others.
SystemLayer::PostEvent add API to post a lambda event #9678
Unify SystemPool and lib/support/Pool #9590
Merge SystemPool and support/Pool #9748

For those, a link in the description is sufficient not a copy of the whole text. But it is definitely a pain that this works so poorly in GitHub.


Introduce assertThatCharSequence to disambiguate Groovy's GString

Feature summary
Currently, Groovy's interpolated expressions cannot be a direct input of assertThat as groovy.lang.GString implements both Comparable and CharSequence.
We might add assertThatCharSequence(CharSequence) / assumeThatCharSequence(CharSequence) and the corresponding BDD variants to disambiguate these calls.
Example
This should compile:
  @Test
  void should_accept_interpolated_expressions() {
    // GIVEN
    def foo = "foo"
    def actual = /.*${foo}.*/
    // WHEN/THEN
    assertThatCharSequence(actual).isEqualTo(".*foo.*")
  }

Implementation notes:

The implementation can delegate to the existing assertThat(CharSequence)
The new method should also be added to Java6 classes whenever possible. Assertions_sync and Java6Assertions_sync can help figure out where the gaps are.

Workaround
Use toString:
  @Test
  void should_accept_interpolated_expressions() {
    // GIVEN
    def foo = "foo"
    def actual = /.*${foo}.*/
    // WHEN/THEN
    assertThat(actual.toString()).isEqualTo(".*foo.*")
  }

Hey @scordio,  I am looking for an issue for my first contribution . Can you help me here. Can I take on this .

Hi @techrajdeep, sure, this issue is up for grabs. Anything where you'd need clarification?

@scordio Could you please assign it to me

Hey @scordio! Has this issue been fixed? I don't see a PR for it. If this is still up for grabs, I'd like to work on this.

Sure and thank you, @debanjanc01!

Hey @scordio is there an existing test case that tests this out as you have mentioned in the example for should_accept_interpolated_expressions? Or shall I add the test case to Assertions_assertThat_with_Groovy_strings_Test.groovy?

Hey @scordio, the problem with groovy.lang.GString is the way it implements the equals(Object) method. It expects that the other object is also an instance of groovy.lang.GString.
In the test case mentioned, assertThatCharSequence(actual).isEqualTo(".*foo.*") will lead to assertion of groovy.lang.GString against java.lang.String. On delegating to the existing assertThat(CharSequence) this would fail as internally CharSequenceAssert#isEqualTo(Object) would call StandardComparisonStrategy#areEqual(Object, Object).
I wanted to know if you have any suggestions to take this forward. This is to understand your preferences and weigh out the different options before proceeding.

Thanks for looking into this, @debanjanc01!
I see the problem, let me think about it and I'll get back to you.

Hey @scordio , wanted to check if you've had the time to look into this.

Hi @techrajdeep, apologies for the terribly late feedback 😅

Hey @scordio is there an existing test case that tests this out as you have mentioned in the example for should_accept_interpolated_expressions? Or shall I add the test case to Assertions_assertThat_with_Groovy_strings_Test.groovy?

There is no existing test case at the moment and Assertions_assertThat_with_Groovy_strings_Test.groovy is definitely the right place for a new one.

Hey @scordio, the problem with groovy.lang.GString is the way it implements the equals(Object) method. It expects that the other object is also an instance of groovy.lang.GString.

After checking the topic again and looking at the Groovy sources, the test case I mentioned was definitely wrong. Instead, we should probably target something like the following:
  @Test
  void should_accept_interpolated_expressions() {
    // GIVEN
    def foo = "foo"
    def actual = /.*${foo}.*/
    // WHEN/THEN
    assertThatCharSequence(actual).isEqualTo(/.*${foo}.*/)
  }

I would still delegate to assertThat(CharSequence). As GString::equals would return false with a String input, I don't see anything wrong if isEqualTo would fail.
Would you still have time to pursue this topic? Otherwise, I'm happy to take it over.

Hey @scordio thanks for taking the time out to revisit this. Let me check this and get back.


[SPARK-32271][ML] Add option for k-fold cross-validation to CrossValidator

What changes were proposed in this pull request?

I have added a method parameter to CrossValidator.scala to allow the user to choose between repeated random sub-sampling cross-validation (current behavior) and k-fold cross-validation (optional new behavior). The default method is random sub-sampling cross-validation.
If k-fold cross-validation is chosen, the new behavior is as follows:

Instead of splitting the input dataset into k training and validation sets, I split them into k folds; for each fold of training, one of the k splits is selected for validation, and the others are unioned together for training.
Instead of caching each training and validation set k times, I cache each of the folds once.
Instead of waiting for every model to finish training on fold n before moving on to fold n+1, new fold/model combinations will be trained as soon as resources are available.
Instead of creating one Future per model for each fold in series, all Futures for each fold & parameter grid pair are created and trained in parallel.
A new Int parameter is added to the Future (now Future[Int, Double] instead of Future[Double]) in order to keep track of which Future belongs to which parameter grid.

Why are the changes needed?

These changes allow the user to choose between repeated random sub-sampling cross-validation (current behavior) and k-fold cross-validation (optional new behavior). These changes:

allow the user to choose between two types of cross-validation.
(If k-fold is chosen) only require caching the entire dataset once (instead of k times in repeated random sub-sampling cross-validation, as it does now).
(if k-fold is chosen) free resources to train new model/fold combinations as soon as the previous one finishes. Currently, a model can only train one fold at a time. If k-fold is chosen, the added functionality will allow the fit to train multiple folds at once for the same model, and, in the case of a grid search, allow it to train multiple model/fold combinations at once, without needing to wait for the slowest model to fit the first fold before moving onto the second.

Does this PR introduce any user-facing change?

Yes. This PR introduces the setMethod method to CrossValidator. If the method parameter is not set, the behavior will be the same as it has always been.
How was this patch tested?

Unit tests will be added.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

This is ready for review!

@viirya

ok to test

Test build #125874 has started for PR 29080 at commit 6dd0a4d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30482/
Test PASSed.

Test build #125874 has finished for PR 29080 at commit 6dd0a4d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125874/
Test PASSed.

@viirya Sorry, can you explain? I don't see how it changes anything, it just allows models from multiple folds to be run in parallel.

Do you read the above too links? The current approach is repeated random sub-sampling validation, this PR changes to k-fold cross-validation.

Yes, I know the difference between the two. I just assumed that MLUtils.kFold was doing the splits according to the k-fold method, not the random sub-sampling method. But I suppose changing the name of that method is outside the scope of what I'm trying to add.
In that case, it seems that I should add an addition method parameter where you can select k-fold or random sub-sampling. If I end up doing that, should I continue with this PR or open a new one?
Thoughts @viirya?

I've done a first pass at including both options. Tomorrow I'll clean it up and add the necessary unit tests.

Test build #125888 has started for PR 29080 at commit a0a41b5.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30496/
Test PASSed.

Test build #125888 has finished for PR 29080 at commit a0a41b5.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125888/
Test PASSed.

Test build #126297 has started for PR 29080 at commit 3bc2762.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30912/
Test PASSed.

@viirya I made the changes and added unit tests. Let me know what you think.

Test build #126297 has finished for PR 29080 at commit 3bc2762.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126297/
Test PASSed.

@viirya I certainly have many situations wherein this new approach is better, which is what caused me to open this pull request in the first place. Would it be helpful if this was labeled an "expert" parameter, like some other classes have?

@viirya While it may not work better if the user has limited resources, it will never work worse, and it introduces a new way to do cross-validation that was previously unavailable.

Can one of the admins verify this patch?

@viirya What's the status of this? Can this be merged?

@HyukjinKwon Tagging you because I saw you approved some recent PRs. I still strongly believe this functionality is worth having. Can we take a look?


Add DamagePoint options to DamageDropoff

Description
When I use the dropoff feature, it affects all features such as head, body, arms, legs and feet, but I only want it to affect a few.
Configuration
in multiple use:
Dropoff:
 - <travel distance>~<one of the bonus damage>(head, body, arms, legs and feet),<one of the bonus damage>...~<damage amount>

Example:
Dropoff:
 - "30~head~-5"
 - "30~body,arms~-15"
 - "30~legs,feet~-25"
#or
 - "30~head~-5"
 - "30~body,arms,legs,feet~-25"

in single use:
Dropoff:
 - <travel distance>~<one of the bonus damage>(head, body, arms, legs and feet)~<damage amount>

Example:
Dropoff:
 - "30~head~-5"
 - "30~body~-10"
 - "30~arms~-15"
 - "30~legs~-20"
 - "30~feet~-25"

to use all:
Dropoff:
 - <travel distance>~all~<damage amount>

Example:
Dropoff:
 - "30~all~-25"

Payment (optional)

[ ] I'm willing to pay for this feature

Since this is an extremely niche feature request, I don't think I'll be adding it without a payment.


Entities wont update

Thanks for your work here, it works mostly very good.
The problem is refreshing the entities to get their new states. Is there a way to control the polling or just trigger a refresh without doing unregister and register again?

It's run on a default scan interval of 1800 seconds, https://github.com/glance-/dhl/blob/master/sensor.py#L38 , but that can be overridden in the configuration:
sensor:
  - platform: dhl
    scan_interval: 90

I think it should be possible to trigger a manual refresh of the state with homeassistant.update_entity to, but it's way to long ago I wrote this to remember.

Thank you very much for your fast reply!

The polling runs on the same homeassistant.util.Throttle scheme as all the other freight integrations I've wrote, and, I haven't gotten any dhl packages in a while, but I know for a fact that the PostNord sensor polling works. I got a couple of packets from them last week and those did update just as they should.
You can turn on debug logging for the component, and then you'll get a dump of the json data from the api on each poll cycle. Check that it actually happens. If not something's broken in your home-assistant setup causing the Throttle not to happen, and if it happens you can check that the updated data is actually there. If the data is actually there, something else might be detonating, which you will probably see in the debug logs then.
If you find something erroring out in the logs, post them here and I'll take a look.

I noted by accident that I actually had a dhl packet on my way, and did some digging.
The polling happened on the selected scan_interval as I thought, but I noted that the forcing of a update didn't work by calling homeassistant.update_entity so I fixed that.
Also I switched to using the statusCode for state on the packages, instead of the more descriptive status filed which was just to long. I'm not shure this was apart of the api when I wrote this code, but hey, looks prettier now.
So, yes, the scan_interval actually works for me.

Hey,
thank you for your work and time, i updated instantly after it came and works perfectly now for me.
Thx for your fast reply! I really like the integration. Unfortunatly theres nothing like this for Hermes, GLS and some other german shipment methods.


How to update the model?

How can I update the model with new data?  I tried pickling the Prophet instance and running fit() on new data but I bump into this "Changepoints must fall within training data" exception.  I'd prefer not to disable the automatic changepoint detection.  Thanks!

Thanks for pointing this out. The Prophet object wasn't really intended to be re-fit. What is happening here is that the changepoints are being automatically set using the first dataset, but then they don't make sense for the second dataset so it is throwing the error.
Is there a particular reason why you want to re-fit an existing Prophet object instead of just instantiating a new one? Are you trying to get the changepoint locations to be consistent across the multiple fits?
I'm inclined to fix this issue by entirely disallowing a prophet model to be fit twice, so want to know if there is a reason for making that a possibility.

I'm only evaluating Prophet right now, but I'm wondering some operational details.  For example, what should I do with a new day's worth of data?  Should I fit that along with all the prior data which could be years?  Seems like a waste of compute resources to re-fit already fit data.  It's probably something silly I'm not understanding.

I see, this is related to #46. Currently you would have to completely re-fit when adding a new day of data.
It does seem wasteful and I like the idea of warm-starting the fit from the current parameters if the object has already been fit to earlier data, although it could potentially be a bit complicated. The main complication is the changepoints and the corresponding fitted rate changes. Imagine we have fit Prophet to data on t=1,...,100 and now we are re-fitting on t=50,...,150.  If we are automatically selecting changepoints, then the changepoints in the 2nd dataset will be very different from those in the 1st. This means that the rate change values from the 1st dataset could potentially be really inappropriate for the second dataset, and this 'warm-start' could actually be a bad initialization that harms the fit. If we keep the changepoints constant when moving from dataset1 to dataset2 then the warm-start would be meaningful, and that would work well if we've just added one day of data. But now suppose that we've been running this for 50 days, each day adding one new day of data. We would need to automatically add additional changepoints as the time series gets longer otherwise the performance would degrade compared to a fresh fit. That's where things get a bit complicated and is why this is on the wishlist, especially since the fitting is pretty fast.

34c0f80684b871d49e2112077e62e253e184d430 raises an exception if fit is called on a fitted model which should make this more clear in the future, until we do get some sort of warm-start update figured out.

Oh, that other issue addresses my question - I'll follow that one.  That error msg makes it more clear.  Thanks!

I think that updating a statistical model for time series is useful when for example you want to compare it with a deep learning model for time series, using rolling window with updated origin. If i want to make a short term forecast, i.e. 30 minutes for one day, it would be unfair for a deep learning model to refit a statistical model and then make the forecast as the deep learning model "sees" only the training data and then is provided with the true values for some given window (t-n, ..., t) to forecast the next 30 minutes.  Any comment on this? Thank you in advance.


Update README.md

Before submitting the PR, please go through the sections below and fill in what you can. If there are any items that are irrelevant for the current PR, remove the row. If a relevant option is missing, please add it as an item and add a PR comment informing that the new option should be included into this template.

All relevant items should be ticked before the PR is merged

Description

[ ] Summary of the changes and the related issue:
[ ] Motivation and context regarding why the change is needed:
[ ] List / description of any dependencies or other changes required for this change:
Fixes an issue in GitHub / Jira:

[ ] Yes: [link to GitHub issue / Jira task ID]
[ ] No

Type of change

[ ] Bug fix

[ ] Breaking: Describe
[ ] Non-breaking

[ ] Documentation
[ ] New feature

[ ] Breaking: Describe
[ ] Non-breaking

[ ] Security Alert fix
[ ] Tests (only)
[ ] Workflow

"Breaking": The change will cause existing functionality to not work as expected.
Checklist:
General

[ ] Changelog: New row added. Not needed when PR includes only tests.
[ ] Code change

[ ] Self-review of code done
[ ] Comments added, particularly in hard-to-understand areas
Documentation update

[ ] Done
[ ] Not needed

Repository / Releases

[ ] Blocking PRs have been merged
[ ] Rebase / update of branch done
[ ] PR to master branch (Product Owner / Scrum Master)

[ ] The version is updated

[ ] I am bumping the major version (e.g. 1.x.x to 2.x.x)

[ ] I have made the corresponding changes to the API version

Checks

[ ] CodeQL passes
[ ] Formatting: Black & Prettier checks pass
Tests

[ ] I have added tests for the new code
[ ] The tests pass

Closing in favor of https://github.com/ScilifelabDataCentre/dds_cli/pull/585


[Feature][python] Add example about multi line shell

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
In all our examples in https://dolphinscheduler.apache.org/python/tasks/shell.html#example are single line. we should add example about how to create shell task in multiple lines
Use case
No response
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

And also string format

better to add parameter for our example

@zhongjiajie sir please assign this issue to me, I will try my best to solve it, can you just summarise what exactly I have to do as some of the links are not updated mentioned in this issue, it would be really helpful.

also can you just summarize what exactly I have to do as some of the links are not updated mentioned in this issue, it would be really helpful.
@zhongjiajie sir please give a little bit brief about it.

also can you just summarize what exactly I have to do as some of the links are not updated mentioned in this issue, it would be really helpful.
@zhongjiajie sir please give a little bit brief about it.

@zhongjiajie ....

@zhongjiajie Sir, as I can understand from whatever information is provided that the DolphinScheduler documentation needs to be updated to include a detailed example illustrating how to create and execute a multi-line shell task within a Python context. This goes beyond the existing single-line examples, emphasizing the need to showcase more complex scenarios where shell commands or tasks in Python might extend over multiple lines of code. Am I right in comprehending this, if not then please specify anything which needs to be rectified along with this.

@zhongjiajie Sir, as I can understand from whatever information is provided that the DolphinScheduler documentation needs to be updated to include a detailed example illustrating how to create and execute a multi-line shell task within a Python context. This goes beyond the existing single-line examples, emphasizing the need to showcase more complex scenarios where shell commands or tasks in Python might extend over multiple lines of code. Am I right in comprehending this, if not then please specify anything which needs to be rectified along with this.

Hi @HarshitNagpal29, sorry for replying late, we should submit the change to https://github.com/apache/dolphinscheduler-sdk-python. for more specific line are those lines https://github.com/apache/dolphinscheduler-sdk-python/blob/e04b836e54fb7bfd96527007cfeaa0fef4444b07/src/pydolphinscheduler/examples/tutorial.py#L51-L54 we should pick one of those task and change them as multiple lines shell example

@zhongjiajie No problem Sir , I have created pull request again in https://github.com/apache/dolphinscheduler-sdk-python as you said, please check it and let me know if I have to make more changes in that.


Support block transfers

Some banks may trigger transfers/transactions as blocks.
Currently it is not possible to handle these transactions independently even though they may be listed separately in the CAMT052/CAMT053 files.
Add support for it

Resolved and released as v2.0.0


[Feature] The Robit Revolution

I'm sure we all love our little Robits, following us around and diligently picking up everything we drop.
But other then that and the crafting options he has, the little guy isn't that usefull.
Wouldn't it be fun to have more uses for our hard working robot friend.
My idea is mainly oriented on Thaumcrafts Golems.
The first thing, would be the building of Robits.
instead of having one type of robot, there could be various different ones, constructed from modular parts.
There can be a Robit with the legs, one with chain drive and maybe a robit with a hover modul.
His head could have various different sensors that help him in his daily tasks.
And his arms could have different tools to perform functions.
Now what could our Robits do with all of that.
They cold do your farmwork.
Transport Items through your base.
Sort items into inventories.
Activate Blocks
Fell Trees
Milk Cows and sheer sheeps
and patrol areas, fighting monsters and maybe even other players.
There can even be some mod compability with a robot that communicates with computercraft or open computers
I do realise that some of these features are already implemented in some other way and some are maybe even planed with some other machines, but it's always fun to have options.
Also, I'm sure it is especially challanging to realize complex features like this with entities.
This little piece of text is just meant to inspire some ideas and I'd love to hear some opinions on that matter.
Give Robit some love.

@EvilCooky said: They cold do your farmwork. Transport Items through your base...
This sounds like the sort of thing I've done in the past with
OpenComputers.  Build and program a robot to mine for you, harvest trees
and sugarcane, etc.  Highly recommended if you want serious flexibility
in getting things automated.

Factorio drones?

More like Thaumcraft golems.

This is a bit too much copying TC. I could understand farming/milking/w/e robits but pipes exist (maybe area pickup robit only?) and they don't look durable enough to fight.


⚠ Secret Website 3 has degraded performance

In 439e78b, Secret Website 3 ($SECRET_WEBSITE_3) experienced degraded performance:

HTTP code: 200
Response time: 2902 ms

Resolved: Secret Website 3 performance has improved in d94162d.


validate_request for RefreshTokenGrantType was rewritten in general form

cc @danygielow

@danygielow good point! I will fix this evening :)


Update sbt, scripted-plugin to 1.5.6

Updates

org.scala-sbt:sbt
org.scala-sbt:scripted-plugin

from 1.4.9 to 1.5.6.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Applied Scalafix Migrations

org.scala-sbt:sbt:1.5.0 (created no change)

https://gist.githubusercontent.com/eed3si9n/57e83f5330592d968ce49f0d5030d4d5/raw/7f576f16a90e432baa49911c9a66204c354947bb/Sbt0_13BuildSyntax.scala
Documentation: https://eed3si9n.com/syntactic-scalafix-rule-for-unified-slash-syntax

Files still referring to the old version number
The following files still refer to the old version number (1.4.9).
You might want to review and update them manually.
CHANGELOG.md

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-sbt" } ]

labels: library-update, early-semver-minor, semver-spec-minor, scalafix-migrations, old-version-remains, commit-count:1

Superseded by #35.


Laravel 11.x Compatibility

This is an automated pull request from Shift to update your package code and dependencies to be compatible with Laravel 11.x.
Before merging, you need to:

Checkout the l11-compatibility branch
Review all comments for additional changes
Thoroughly test your package

If you do find an issue, please report it by commenting on this PR to help improve future automation.

:warning: The latest version of Laravel has a dependency for phpunit/phpunit of ^10.5. If you would like to also upgrade your tests, you may run the PHPUnit 10 Shift for free.

:alembic: Using this package? If you would like to help test these changes or believe them to be compatible, you may update your project to reference this branch.
To do so, temporarily add Shift's fork to the repositories property of your composer.json:
{
    "repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/laravel-shift/laravel-eloquent-flag.git"
        }
    ]
}

Then update your dependency constraint to reference this branch:
{
    "require": {
        "cybercog/laravel-eloquent-flag": "dev-l11-compatibility",
    }
}

Finally, run: composer update


Specify dialect per external model

Issue
As per the duckdb example configuration (https://sqlmesh.readthedocs.io/en/stable/integrations/engines/duckdb/?h=duck#duckdb-catalogs-example) it is possible to attach a variety of data catalogs, however if specifying these as external tables there is no way to specify the dialect for each external table - ie. you may have an external model sqlite.main.example and another model postgres.public.example2, which should each be specified as per their dialect.
Currently the only way to specify the dialect for external tables is via model_defaults.
gateways:
  my_gateway:
    connection:
      type: duckdb
      catalogs:
        memory: ':memory:'
        sqlite:
          type: sqlite
          path: 'test.db'
        postgres:
          type: postgres
          path: 'dbname=postgres user=postgres host=<IP_ADDRESS>'
          read_only: true

completed here - https://github.com/TobikoData/sqlmesh/pull/3136


Add configurable request timeout options

I would like to be able to adjust the timeout time so that it can be dynamically changed if there is a large delay or host latency.
@hakanensari, check please

@gogochikago good proposal. not sure that I agree with proposed implementation through, but before giving my comments on that... I wanted to ask.
If I remember correctly, you can set default timeouts for excon for yourself without modifying this gem. Like this:
Excon.default[:connect_timeout] = 5
Excon.default[:read_timeout] = 60

Have you tried that? Is there any reason why that doesn't work for you?

To add to @skatkov's comment
Correct me if I'm wrong, but the defaults of Excon for both values seem to be 60 seconds. If so, overriding the default read_timeout would be redundant?


Error

It looks like the plugin is looking to connect to port 80 instead of 8080 when requesting additional endpoints.  The main endpoints work fine.  Either that or it's because of different Sonnen firmware versions.  Not sure.  Error fills up the logs.
sonnen.0 | 2020-05-15 17:40:25.203 | warn | (958) [ADDITIONAL] Error on requesting additional endpoints: RequestError: Error: connect ETIMEDOUT XX.XX.XX.XX:80
Kevin
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots & Logfiles
If applicable, add screenshots and logfiles to help explain your problem.
Versions:

Adapter version:
JS-Controller version:
Node version:
Operating system:
Battery model:

Additional context
Add any other context about the problem here.

One request correctly goes to port 80 introduced with the latest update. So I guess the error occurs at every poll?

What happens when you try to open <battery-ip>/api/online_status in your browser.

please try 1.5.1

It occurs every poll because I don’t have port 80 forwarded to the Sonnen. I did not know port 80 was even being used on the Sonnen. I may use port 80 for another device.
Sent from my iPad

On May 16, 2020, at 3:12 AM, Max Hauser<EMAIL_ADDRESS>wrote:

One request correctly goes to port 80 introduced with the latest update. So I guess the error occurs at every poll?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

80 is standard http port ..


Support grouping sets

(you don't have to strictly follow this form)

Use case

A clear and concise description of what is the intended usage scenario is.

Describe the solution you'd like
Spark support grouping sets,  https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-groupby.html.
The phyical plan for running grouping sets at present is like following
CHNativeColumnarToRow
+- *(5) SortExecTransformer [msgtype#114L ASC NULLS FIRST, hour#115 ASC NULLS FIRST], true, 0
   +- ColumnarExchangeAdaptor rangepartitioning(msgtype#114L ASC NULLS FIRST, hour#115 ASC NULLS FIRST, 5), ENSURE_REQUIREMENTS, false, [plan_id=132], [id=#132], [OUTPUT] List(msgtype:LongType, hour:IntegerType, count(1):LongType), [OUTPUT] List(msgtype:LongType, hour:IntegerType, count(1):LongType)
      +- *(4) HashAggregateTransformer(keys=[msgtype#114L, hour#115, spark_grouping_id#113L], functions=[count(1)], output=[msgtype#114L, hour#115, count(1)#110L])
         +- ColumnarExchangeAdaptor hashpartitioning(msgtype#114L, hour#115, spark_grouping_id#113L, 5), ENSURE_REQUIREMENTS, false, [plan_id=128], [id=#128], [OUTPUT] List(msgtype:LongType, hour:IntegerType, spark_grouping_id:LongType, count:LongType), [OUTPUT] List(msgtype:LongType, hour:IntegerType, spark_grouping_id:LongType, count:LongType)
            +- *(3) HashAggregateTransformer(keys=[msgtype#114L, hour#115, spark_grouping_id#113L], functions=[partial_count(1)], output=[msgtype#114L, hour#115, spark_grouping_id#113L, count#120L])
               +- RowToCHNativeColumnar
                  +- *(1) CustomExpand [[msgtype#95L, hour#109, 0], [msgtype#95L, null, 1], [null, null, 3]], [msgtype#114L, hour#115, spark_grouping_id#113L], [msgtype#114L, hour#115, spark_grouping_id#113L]
                     +- CHNativeColumnarToRow
                        +- *(2) ProjectExecTransformer [msgtype#95L, hour#109]
                           +- FileScan parquet [msgtype#95L,day#108,hour#109] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/github/gluten-docker/packages/like_all_push_hour_test2], PartitionFilters: [isnotnull(day#108), (day#108 = 2022-09-16)], PushedFilters: [], ReadSchema: struct<rip:string,rtime:string,appid:bigint,svrid:bigint,stattime:bigint,msgid:string,msgtype:big...

We need to support the columnar expand operation.

A clear and concise description of what you want to happen.

Describe alternatives you've considered
There have been a trasformer ExpandExecTransformer for CustomExpand, but is not enable for ClickHouse.
We need to add a new plan parser for substrait expand node.

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

closed by https://github.com/Kyligence/ClickHouse/pull/251


"HTTP Error 502.5 - Process Failure" asp.net core error in Azure From VSTS

I have a problem with deploying from VSTS to Azure Web App for NET CORE 2.0 app (my app use 2.0.6.). I get the well known HTTP Error 502.5 - Process Failure. Funny thing is that it works if I deploy from Visual Studio. And it's installed in West Europe region.
The Kudu shows this:

D:\home\site\wwwroot>dotnet --info
.NET Command Line Tools (2.1.101)
Product Information:
Version:            2.1.101
Commit SHA-1 hash:  6c22303bf0
Runtime Environment:
OS Name:     Windows
OS Version:  10.0.14393
OS Platform: Windows
RID:         win10-x86
Base Path:   D:\Program Files (x86)\dotnet\sdk\2.1.101\
Microsoft .NET Core Shared Framework Host
Version  : 2.0.6
Build    : 74b1c703813c8910df5b96f304b0f2b78cdf194d

If I try to run:

D:\home\site\wwwroot>dotnet My.API.dll
Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at My.API.Program.BuildWebHost(String[] args) in D:\vsts-agent-win-x64-2.129.1_work\1\s\My.API\Program.cs:line 15
at My.API.Program.Main(String[] args) in D:\vsts-agent-win-x64-2.129.1_work\1\s\My.API\Program.cs:line 1

I tried re-deployment, compared web.config, deps.json file, appSettings.json and all looks ok. Any ideas?

Have you enabled the app_offline rule?

I have this in my deploy:

The fact you don't have the Publish using Web Deploy chosen is throwing me off. Are you using WebDeploy?

I turned it ON, but still no luck. I compared the two zipps, and the one build with VSTS has more DLLs in than the one from VS2017.
Here is my publish step:

Just fixed this issue. I have to add "Path to projects(s)" csproj file of the exact project. Then it worked, because all test projects were not included in the publish!


refactor: Use DSL v2 types in place of ScannerDefinitionNode

Based on #1002
Part of #638
The DSL v2's model::Scanner is almost 1:1 usable with our current parser codegen, with the exception of versioned scanners. In short we:

introduce a helper type VersionedScanner that is used in place of the old ScannerDefinitionNode::Versioned
introduce a ScannerExt trait implemented for both VersionedScanner and the model::Scanner, which is responsible for generating the main scanning logic
Repurposes ScannerDefinition slightly to surface more of the scanner-related logic for trivia/fragment/token
implements this directly for model::{TriviaItem,FragmentItem,TokenItem} and stores it in the v1 Grammar struct
similarly, uses model::KeywordItem directly in place of v1's KeywordScannerDefinition as they shared the same functionality

@AntonyBlakey I've changed some names, let me know if that's okay now 🙏

Not 100% happy with the naming in this section of our codebase, but this PR is not the place for that more general argument, so I am approving this now.


Any plans to support .ignore?

I've experienced hard to ignore folders without synchronization.
I use oneDrive for source repository. When I build my projects, oneDrive client try to synchronize thousands of files. So I unchecked build and target folders in Settings, remove them in online site. But after building project, oneDrive client alerts Synchronization Problem with red exclamation mark.
I really hope oneDrive support .ignore file like .gitignore

Sounds like you're talking about the OneDrive sync client. Since this is a feature suggestion, can you add it on the onedrive uservoice page: https://onedrive.uservoice.com/forums/262982-onedrive

For anyone else finding this from a Google search, this is the issue that needs your votes: https://onedrive.uservoice.com/forums/913522-onedrive-on-windows/suggestions/18449926-reassess-the-possibility-to-ignore-or-exclude-sele

For anyone else finding this from a Google search, this is the issue that needs your votes: https://onedrive.uservoice.com/forums/913522-onedrive-on-windows/suggestions/18449926-reassess-the-possibility-to-ignore-or-exclude-sele

Unfortunately, the link is broken.  (404) "UserVoice instance is no longer available."

For anyone else finding this from a Google search, this is the issue that needs your votes: https://onedrive.uservoice.com/forums/913522-onedrive-on-windows/suggestions/18449926-reassess-the-possibility-to-ignore-or-exclude-sele

Unfortunately, the link is broken. (404) "UserVoice instance is no longer available."

Yes, Microsoft recently decided to delete all of their UserVoice instances without notice and is thinking about making the top requests available again somewhere else. Most user request and comments, including those shared at that link, have been deleted forever.
Don't ask me why they did this, the only reason that makes any sense is that they decided user's voices don't actually matter after all and they're tired of us asking for the features that we need.

I just found out today that Microsoft has created their own portal to replace UserVoice and has imported the most popular requests from UserVoice as indicated. Unfortunately, no comments have been migrated and likely never will.
The issue linked above can now be found here: https://feedbackportal.microsoft.com/feedback/idea/084d42ab-d51c-ec11-b6e7-0022481f8143

Microsoft rolls out function for excluding files, but there is no support for excluding folders right now. Please check the link below:
https://docs.microsoft.com/en-us/onedrive/use-group-policy#exclude-specific-kinds-of-files-from-being-uploaded)

any updates on a issue request somewhere on Microsoft's side to request folder exclusion?

@hotcuppajojo a policy was added to the docs recently, and in the binary for a while, but I can't actually get it to work. regkeys are loaded but the folders are still synced

@hotcuppajojo a policy was added to the docs recently, and in OneDrive.exe for a while, but I can't actually get it to work. regkeys are loaded but new matching folders are still synced. tested with Insider and Internal Fast (24.078.0416.0001)

I also tried to get this working and failed. However, I also admit that I don't really know what I'm doing lol. Here's what I tried though, in case it helps. I didn't see the key HKLM\SOFTWARE\Policies\Microsoft\OneDrive\EnableODIgnoreFolderListFromGPO in the registry, so I tried adding it, and added the value .Rproj.user hoping that all folders named .Rproj.user would be ignored, but the folders are still syncing.

any news about exclude folders? The file have exclude successfully by "EnableODIgnore", but it's failed for subfolders.

it was recently removed from the docs, but codepaths still exist in OneDrive.exe. I've been tracking it here https://garden.tplant.com.au/microsoft/one-drive/


Remove debug prints from manage.py

Remove a couple of print statements from manage.py.

Codecov Report

Merging #558 (b2e7632) into master (27e7798) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #558      +/-   ##
==========================================
- Coverage   92.68%   92.67%   -0.02%
==========================================
  Files         319      319
  Lines       19676    19676
  Branches     1573     1573
==========================================
- Hits        18236    18234       -2
- Misses       1144     1145       +1
- Partials      296      297       +1

Impacted Files
Coverage Δ

importer/taric.py
78.28% <0.00%> (-1.32%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 27e7798...b2e7632. Read the comment docs.


Question about systray

Do I apply the systray patch on the suckless website then apply the one in this repo, or does the one in this repo alone does the job?

The README clearly mentions what you need to do. Please read that.
patches folder contains two patches for dwm, ONE FOR DWM ALREADY
PATCHED WITH SYSTRAY PATCH and the other for vanilla dwm. One of the patches,
whichever appropriate, is essential for dwmblocks to function properly. It will
add support for colored text, clickability and cursor hinting when hovering on
clickable blocks (inspired by polybar).

Ok, thanks. Sorry for asking question without doing properly reasing first.

Ok I applied it and it worked perfeclty. Another question though, is there any way to have the systray to the left of dwmblocks? I see that the diff removed that option from config.def.h, so there is a reason probably.

I didn't have much time to look into it, so I did the simplest thing possible. I will look into integrating it later some time (I thought it wasn't that useful).

Ah, ok, understandable.


docs: Add samples for variables in nested loops

Fixes [b/207785641]

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

Will use valid filenames


txt2al formatting <> VS Code AL autoformatting

I noticed that the VS code autoformat feature (it seems to kick in when the AL language extension recognizes a change to a file that's been modified+saved) produces a format that, while harmless (i.e. adding spaces after semicolons, etc), it produces an .al file that results in possibly hundreds of diffs from the originating file that was outputted via the txt2al tool, thus adding unnecessary noise to my git workflow.
it would be nice if the AL Extension automatter and txt2al formatting were identical. Currently they are not.

+1, would also like if the keywords and calls to built-in functions would comply with the new conventions/standards (most of them will still be in ALLCAPS).

And please add pair of brackets if missing:
INIT; => Init();

I also noticed that the BC sandbox web client sandbox page designer does not format .al files in accordance to VS Code's AL auto-formatting (i.e. indendation different)

Starting with Spring 2019 CU3 the Txt2Al tool will automatically format the file using the same formatting engine as the AL compiler. I will create an item to track formatting the code generated by the designer.
For changes to the syntax that should be performed automatically, and which are not exclusively whitespace related, please open another issue.


broke link

another broken link https://autodistill.github.io/autodistill/base_models/detr/

This issue has now been fixed. Thank you for reporting!


convert "Represent quantum computers" from MDX to Jupyter notebook

part of #169

Running into https://github.com/Qiskit/qiskit/issues/11998

Can you delete any now unused images?

The stale image script wasn't returning anything and I realized the reason is that the stale translation pages still use them. In previous PRs, I may have removed some images without checking whether they were used in the translations. What should we do?

I'm removing the stale translations in https://github.com/Qiskit/documentation/pull/1040.

when you're replacing old files with new jupyter notebooks can you make sure to add the original authors as co-authors for your commit?

@javabster I did do that. See https://github.com/Qiskit/documentation/pull/1018/commits/3f1b3469b462d69932f5b589fd4256ccedf33681.


[Patch Request] Cave Noire (Gameboy)

Requesting a patch for Cave Noire for the Game Boy. It's a turn-based strategy game...almost better if no one makes a patch so I'm not tempted to cheat by save scumming 😶🌫

Hi,
I have created a patch for this game (both Japanese and English patched version). Let me know if any issues.


fix: use CBO rule instead of using rewrite rule.

Proposed changes
Issue Number: close #xxx
Problem summary
Currently, rewrite will use pattern any(), it will cause performance regression.
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


Draft: [ext] 🛠 Fixed docker-compose.yml

Problem:
After version update with docker distributed label-studio, there was no actual update 😅.
Because the logic files had been fixed in this volume and were not updated.
Solution:
Removing volume
- static:/label-studio/label_studio:rw

Fixed in https://github.com/heartexlabs/label-studio/pull/1363


petsc: 3.7.6 -> 3.8.3

Semi-automatic update. These checks were performed:

built on NixOS
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/saws/getSAWs.bash -h got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/saws/getSAWs.bash --help got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/saws/getSAWs.bash help got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py -h got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py --help got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py help got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py -V and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py -v and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py --version and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py version and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py -h and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py --help and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/update.py help and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/petscnagupgrade.py -h got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/petscnagupgrade.py --help got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/petscnagupgrade.py help got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/sendToJenkins -h got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/sendToJenkins --help got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/sendToJenkins -h and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/sendToJenkins --help and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/petscdiff -h got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/petscdiff --help got 0 exit code
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/petscdiff -h and found version 3.8.3
ran /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3/bin/petscdiff --help and found version 3.8.3
found 3.8.3 with grep in /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3

Failure on x86_64-darwin (full log)
Partial log (click to expand)
  File "/private/tmp/nix-build-petsc-3.8.3.drv-0/petsc-3.8.3/config/BuildSystem/script.py", line 286, in executeShellCommand
    checkCommand(command, status, output, error)
  File "/private/tmp/nix-build-petsc-3.8.3.drv-0/petsc-3.8.3/config/BuildSystem/script.py", line 239, in defaultCheckCommand
    if status: raise RuntimeError('Could not execute "%s":\n%s' % (command,output+error))
RuntimeError: Could not execute "/usr/bin/install_name_tool -id /nix/store/lvd74a27sknazdah9jas5jdkb2lxbv89-petsc-3.8.3/lib/libpetsc.3.8.dylib /nix/store/lvd74a27sknazdah9jas5jdkb2lxbv89-petsc-3.8.3/lib/libpetsc.3.8.3.dylib":
xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'.

make: *** [makefile:280: install] Error 1
builder for '/nix/store/6w053cdkddjmcxpdbxkd9598y8zs6dc4-petsc-3.8.3.drv' failed with exit code 2
error: build of '/nix/store/6w053cdkddjmcxpdbxkd9598y8zs6dc4-petsc-3.8.3.drv' failed

Success on x86_64-linux (full log)
Partial log (click to expand)
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
checking for references to /tmp/nix-build-petsc-3.8.3.drv-0 in /nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3...
/nix/store/fgr3yvhrnvfhr0czc4ygx1mnjsrv4f4i-petsc-3.8.3

Failure on aarch64-linux (full log)
Partial log (click to expand)
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/79yn486wpiljbnkyl4kwkn5x8xj90ll8-blas-3.7.1
shrinking /nix/store/79yn486wpiljbnkyl4kwkn5x8xj90ll8-blas-3.7.1/lib/libblas.so.3.7.1
strip is /nix/store/viv0il4zpdqj4p7yamwbmn2k9pb8dcnv-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/79yn486wpiljbnkyl4kwkn5x8xj90ll8-blas-3.7.1/lib
patching script interpreter paths in /nix/store/79yn486wpiljbnkyl4kwkn5x8xj90ll8-blas-3.7.1
checking for references to /build in /nix/store/79yn486wpiljbnkyl4kwkn5x8xj90ll8-blas-3.7.1...
cannot build derivation '/nix/store/jglsfw1hn9y10nn3s1makxz03b43rd3x-petsc-3.8.3.drv': 1 dependencies couldn't be built
error: build of '/nix/store/jglsfw1hn9y10nn3s1makxz03b43rd3x-petsc-3.8.3.drv' failed


finish first implementation of the legacysurvey module

To understand:

necessary parameters for the file to be requested during an object/tractor query
necessary parameters for the file to be requested during a brick_list query
necessary parameters fr the request of a region query

* necessary parameters for the file to be requested during an object/tractor query

* necessary parameters for the file to be requested during a brick_list query

* necessary parameters fr the request of a region query

Closed by mistake.
Re-opened

A first implementation in the legacysurvey is now completed.
Focus will now be on the renku plugin development


roswell: <IP_ADDRESS> -> <IP_ADDRESS>

Users may have to delete a file ~/.roswell/env/roswell/impls/x86-64/linux/sbcl-bin/system/dump/roswell.core to make roswell rebuild its core for a newer sbcl version.
This happens every time sbcl is updated. How is to be dealt with that? (I could wrap ros to print a message to that effect.)
Motivation for this change
Update roswell.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

bump :)

bump

bump

This happens every time sbcl is updated. How is to be dealt with that? (I could wrap ros to print a message to that effect.)

yeah, that would be a good idea. Also undraft the PR when you are ready.


Getting a THREE is not defined error

Just installed this now. I'm using webpack to packaged everything up.
I've got the following setup at the top of my file
var THREE = require('three');
var buffer = require('three-buffer-vertex-data');

I'm getting a THREE is not defined error from line 62 of three-buffer-vertex-data. Any ideas why i might be getting this error?
Cheers.

Hey, unfortunately this module relies on THREE being in global scope. I have explored a couple alternative approaches in modules; you can see some more discussion on it here.
https://github.com/mattdesl/three-orbit-controls/issues/17#issuecomment-233139027

Hey @mattdesl, thanks for the quick response!
So if i was to pass my instance of THREE in to three-buffer-vertex-data like so:
var buffer = require('three-buffer-vertex-data')(THREE);

That would be acceptable? Is this currently module setup to accept this currently via commonjs, or would i have to modify it?

Unfortunately, this module just assumes THREE exists globally.
Typically when working with ThreeJS in production and demos, I just end up doing something like this in my source root:
global.THREE = reqiure('three');

This allows me to easily replace three with a local file (like require('./custom-three.js')) or a fork (like require('@jam3/three')), or even just use the script tag approach like this:
<script src="three.js"></script>
<script src="bundle.js"></script>

IMHO there isn't really a solid approach yet as ThreeJS is still discovering modularity and npm. :smile:


Friendly to Designers Alike

Please make bootstrap more friendly to designers alike. Hopefully in v2.0
e.g.:
/less/bootstrap/topbar.colors.less
/less/bootstrap/topbar.skeleton.less
/less/bootstrap/popup.colors.less
/less/bootstrap/popup.skeleton.less
....
....

or, which 1 file includes constant colors
/less/bootstrap/topbar.less
/less/bootstrap/popup.less

or an overrides:
/less/bootstrap/default-theme.less

Please? :)

How does this help?

It helps, it differentiate each of the things like topbar, popovers, forms, it differentiate structure and theming to it.  In current bootstrap, the variables.less must hold the coloring stuff, instead now you have to browse to all files and modify it.. sure i can do a change it, but it would be great if its already there, isn't it?

I'm afraid we won't be taking that approach to Bootstrap. We may make things more granular, but this takes it a bit too far :). As we continue development, we'll try to improve the location of certain blocks of code to make things easier on all.
Thanks for the feedback!


Add support for interpolated docstrings

To simplify writing rule docstrings, and keeping common argument docstrings
consistent, here we add support for docstring interpolation. An example
use-case is a suite of rules (e.g.: foo_library, foo_binary, foo_test) that
all share some common arguments. Rather than copy/paste the argument
docstrings for each rule, adding support for string interpolation simplifies
writing docs and helps keep them in sync.

Can one of the admins verify this patch?

I signed it!

This issue came up while experimenting with generating docs for the Scala rules. In particular, there are a number of common args:
https://github.com/bazelbuild/rules_scala/blob/master/scala/scala.bzl#L510
This implementation is a sketch to run the idea by you. If you like this general concept I can add docs for this new convention, add a test, etc.

@traviscrawford - Sorry for the delay.
This is certainly an interesting idea. Do any of Python's docgen frameworks have a similar feature? Also, do you have an example of what the docstrings would look like with docstring interpolation?


Move "Report" feature to inside of app?

The Issue
The Report feature, which is used to report content for infringement, opens in a separate web page outside of the app.
Should this be moved to inside the app eventually?
System Configuration

LBRY Daemon version:
LBRY App version:
LBRY Installation ID:
Operating system:

Anything Else

Screenshots

I think this should definitely live inside the app. Shouldn't be too difficult.

Working on this.

@seanyesmunt @tzarebczan I have an implementation ready, but it's missing the actual submitting of the report. I couldn't figure out how the current web form does the submitting, and there's nothing in the app for this yet.
How this implementation works is:

Click the report button on a file page
App shows report page and prefills the uri of the file
Fill in the form and submit
App shows a confirmation with snackbar and navigates back to the file page

This can also be used by itself, for example from the sidebar. Only then the infringing file uri would not be prefilled.

@miikkatu thanks for working on this. Unfortunately, the current end behavior of a DMCA report is just trigger an email (see here) and this is not easy to just move into the app.
I filed an issue in our internal-apis project to build an endpoint for receiving these, but I'm afraid this one is blocked until that happens.
Although I suppose you could post to lbry.io/dmca in the interim... that would allow this to be shipped.

Although I suppose you could post to lbry.io/dmca in the interim

I was thinking about this too. For this, we would need an URL where to POST, and an object format that it would accept. I couldn't figure those out in the lbry.io/dcma code. Then, a team member could verify if the posted data is proper or not.
Otherwise, we could wait with this until there's an endpoint available. 🙂

@miikkatu this is what the post call looks like - should be able to pass the same parameters (might need to URL encode them as well)

@miikkatu we'll likely get this fixed web-side within a week or so. I'm inclined to just allow unrestricted POSTs to lbry.io/dmca.
See https://github.com/lbryio/lbry.io/issues/517

PR has been submitted on lbry.io https://github.com/lbryio/lbry.io/pull/525

@miikkatu CORS change is merged if you want to test this in app. Feel free to spam the endpoint, they only come to a few people and they'll be aware. (@finer9 @tzarebczan)

Hmm, @maximest-pierre any thoughts? It was merged here: https://github.com/lbryio/lbry.io/pull/525

Yes it was merge, I look at it quickly this morning. Both GET and POST header shows that its enabled.

Is it still a 405 error or is it a 301 error now. @miikkatu

@maximest-pierre 405, just like before (screenshot above).

Can you open a PR with your current code so I can recreate your error.

@maximest-pierre PR opened. The fetch request code is in src/renderer/page/reportcontent/view.jsx. Hopefully you can recreate the error and find out what's going on. 🙂

closing, we will keep the webpage and not move into the app


Switch main.scss -> main.css

This is a minor change, but enables a user to cd app and then run any server (without needing to npm install anything) to play around with Web Starter Kit.
Simply renaming the file back to main.scss or dropping in other .scss files will still work fine in the styles directory, but figured this was worth making a change for.
r: @sindresorhus or @passy

:+1:

Cool! :+1:

Thanks fellas!