implement pause/resume to RequestQueue?

I'm not sure if it's possible, but would be awesome if you could create a requestQueue in a paused state, then later on, resume it, akin to streams pause()/resume() (only the naming, not that RequestQueue is a stream, but more of a iterator). sometimes you want to enqueue a really big list of items that takes ages to parse, just to get the thunder stolen by other concurrent requests as you're doing a await requestQueue.addRequest().
this is becoming very clear when parsing sitemaps that contains thousands of ready to use items, but you can't just finish enqueuing everything before your desiredConcurrency of 80 kicks in and start doing all the other requests like crazy on the platform.
semi pseudocode:
const requestQueue = await Apify.openRequestQueue();
const requestList = await Apify.openRequestList();

await requestQueue.pause(); // autoscaledPool will not try to "fetchNextRequest()" in a paused state

new Apify.CheerioCrawler({
    requestList,
    requestQueue,
    handlePageFunction: async ({ request }) => {
        const { label } = request.userData;

        if (label === 'start') {
            for (const url of parseSitemap()) {
                await requestQueue.addRequest({
                    url,
                    userData: {
                        label: 'detail'
                    }
                })
            }

            await requestQueue.resume(); // autoscaledPool can now start fetchNextRequest'ing
        } else if (label === 'detail') {
            // do detail
        }
    }
})

Couldn’t the same be achieved with [autoscaledPool.pause and resume] (https://sdk.apify.com/docs/api/autoscaled-pool#autoscaledpoolpausetimeoutsecs)?
Also with custom implementation of AutoscaledPoolOptions.isTaskReadyFunction
The thing is, the queue really is just a data structure. It doesn’t do anything on its own. So in theory, you should not be able to pause or resume it, because there are no actions taken by the queue. Unlike streams, which emit data events, you need to query the queue to get data.

the issue is, if there's a timeout on the current handlePageFunction, it will be stuck in pause (since you can only access it from the inside). it's almost the same issue with streams that don't produce, it needs a lot of boilerplate with BasicCrawler to be able to safely do this, so I thought it could be done on the queue itself, regardless of the autoscaledpool activity.
the issue is even more pronounced when there's a migration happening, and it starts over, but already have some enqueued items...

What do you mean by "you can only access it from the inside"?
You can use crawler.autoscaledPool to access it, no?
I've never tried it, but I'm quite certain you can pause it even before the first request starts processing
const runPromise = crawler.run() // need this to create an instance of AutoscaledPool
await crawler.autoscaledPool.pause();
await runPromise;

the RequestQueue is iteratoresque (fetchNextRequest()) I thought it would make sense to add pause/resume there.
I tried setting the maxConcurrency to 1 at instantiation, then changing the maxConcurrency inside the handlePageFunction, didn't do too well, for the issues mentioned earlier.
Another case is that I have a sitemap that have inner sitemaps, that need to be parsed before "fetching next" the queue while still adding items to it, starting from the RequestList.
I can see isTaskReadyFunction working, but it seems that I can't override it in any crawler, per docs. it would need to be implemented (as in, call the internal one after calling the provided one on class instantiation). or hack my way through, saving then overriding the _isTaskReadyFunction with my own in the internal BasicCrawler, and doing the check for the "go signal" on a variable outside, then calling the saved reference later.
If I could subclass (worst case scenario) both RequestQueue and AutoscaledPool, and provide them as constructors for the crawlers (like new Apify.BasicCrawler({ autoScaledPoolConstructor, requestQueueConstructor }), would "suffice", but again, I'm trying to avoid hacking the SDK or trying workarounds, as mutating/attaching class instances are just too hacky

the RequestQueue is iteratoresque (fetchNextRequest()) I thought it would make sense to add pause/resume there.

Yes, you could think of it as an iterator. But there are no .pause() or .resume() functions in an iterator. There's only .next() and you pause it by not calling next. Same with RequestQueue you pause it by not calling .fetchNextRequest(). So the pausing needs to be implemented on the side of the queue consumers, not the queue itself.
I still don't understand your usecase. Could you explain it without the implementation details?
Eg.: I'm trying to

fetch a sitemap
wait for it to fully download and parse
start scraping the items

or

fetch a sitemap
start downloading and parsing it
while this is going on, start scraping the new requests

or something else?

the real case:

Start it with a maxConcurrency of 1, with a RequestList with one item, sitemap_index.xml can be parsed alone for the sub-sitemaps. Each sub-sitemap is added using requestQueue.addRequest. The crawler won't start executing those because maxConcurrency is 1. After all sub sitemaps were enqueued, set maxConcurrency to like 80, inside handlePageFunction.
Sub-sitemaps now start getting parsed, 3-4 at once, up to 30. each of them use a significant amount of memory (9MB transfer, 260MB in memory), while parsing, each of them are calling requestQueue.addRequest too. Because the concurrency is going up, and I have no control over the queue, the "DETAIL" of each enqueued request start executing at the same time as the parse.
Enqueuing now takes much longer, and now handlePageFunction starting timing out. The same sub-sitemap now needs to be reparsed, but now it's at the bottom of the queue, and 1000's items in front of it.  Memory was already near the max (4GB), with the new DETAIL requests being executed, the outcome is unpredictable.
When the enqueued details are happening, 10 sub-sitemaps that failed earlier starts re-parsing and re-enqueuing items, the crawler fails with OOM, needing to restart it, that does everything again.

so what I wanted to do is: keep parsing sitemaps, if there's ANY sitemap being parsed, no matter how many, I want to "stop" the queue from returning new requests on fetchNextRequest() so autoscaledPool don't call new handlePageFunction with DETAIL until is "unpaused". so I think indeed isTaskReadyFunction should be the proper way to do this, but can't be done without hacking

Ok, this is a bit tricky, but I think the cleanest solution is to

create a temporary named queue, put the first url there,
run it with basic or cheerio, whatever is better for parsing the sitemaps,
collect the sitemap urls into this temp queue and detail urls into the default queue and then,
when the parsing and enqueueing is over, delete the temp queue and launch a second crawler which will crawl the detail (default) queue.

If you really need to optimize this so that the crawling starts as soon as the sub-sitemaps are enqueued (and not yet parsed), which I don’t recommend because it’s difficult enough for the 4gb container with 1 cpu core to parse the 30 sitemaps single-threaded, you can start the other crawler earlier. I’d parse it first and crawl it later. The autoscaled pool will make sure it’s parsing as many sitemaps in parallel as the cpu can handle, so there’s really no extra space for crawling.

I only read this briefly, so maybe I'm missing something. But why do you need to use request queue for adding the URLs from the sitemaps? Can't you just process all the sitemaps and create a RequestList from them, and then starting crawling as usual?

@mnmkng thanks for the idea, I'll split the tasks
@jancurn there's one master sitemap that has link for other sitemaps. by sitemap spec (and per google recommentation), it must not be bigger than 10MB in size and should not contain more than 50k urls. so some sites split them in multiple files, and I need to fetch it, gunzip it, parse, then enqueue

I see, in that case, I'd just process and prepare the sitemaps in memory and then create a RequestList from them. Or create two crawlers, first one for sitemaps, and once it finishes then run a second one for the final URLs.

I'll close this, since we will not be implementing requestQueue.pause()/resume(), but it was a good discussion.

having two basic crawlers in the same actor isn't bad at all at the end, it's much faster and reliable now


Support etcd+TLS

We should have options for both "bring your own etcd" and "self-hosted" etcd. And this should be the default installation method.

The etcd TLS assets needed by the api-server should be populated in a secret
The data that needs to end up on etcd nodes (non-self-hosted) will need to be plumbed through all hack/* examples (e.g. when launching etcd-member.service, it needs to have the generated TLS assets available).
self-hosted etcd will likely require changes to the etcd-operator - as that is responsible for launching the etcd pods - and we would need to add secret volumes to distribute the TLS assets.

Related:
https://github.com/kubernetes-incubator/bootkube/issues/244
https://github.com/kubernetes-incubator/bootkube/issues/389

Related: https://github.com/coreos/tectonic-forum/issues/86

@aaronlevy @mfburnett
Are there any well-defined use case for TLS setup? How does user want to generate certs?
Implementing TLS is simple, but is also permanent.

@aaronlevy  and @mfburnett ... I attempted to enable auto-tls in etcd3  within the tectonic-controller.yaml.tmpl  located in the published Ignition assets from the installer to Provisioner:
systemd:
  units:
    - name: etcd-member.service
      enable: true
      dropins:
        - name: 40-etcd-cluster.conf
          contents: |
            [Service]
            Environment="ETCD_IMAGE_TAG=v3.1.2"
            {{- if index . "external_etcd"}}
            ExecStart=
            ExecStart=/usr/lib/coreos/etcd-wrapper gateway start \
                --listen-addr=<IP_ADDRESS>:2379 \
                --endpoints={{.etcd_endpoints}}
            {{- else}}
            Environment="ETCD_LISTEN_CLIENT_URLS=https://<IP_ADDRESS>:2379"
            ETCD_INITIAL_CLUSTER="node0=https://etcd0.seedbloom.it:2380,node1=https://etcd1.seedbloom.it:2380,node2=https://etcd2.seedbloom.it:2380
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://{{.domain_name}}:2379"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://{{.domain_name}}:2380"
            Environment="ETCD_LISTEN_PEER_URLS=https://<IP_ADDRESS>:2380"
            Environment="ETCD_NAME={{.etcd_name}}"
            Environment="ETCD_AUTO_TLS=true"
            Environment="ETCD_PEER_AUTO_TLS=true"
            Environment="ETCD_STRICT_RECONFIG_CHECK=true"
            {{- end}}

etcd3 cluster is healthy and nodes are talking to each other. However, there is a problem with Kubernetes and publishing information to the controllers. Spent quite a bit of time on this to no avail, we're waiting on this feature to ship instead as it seems Tectonic isn't purposed yet for etcd over TLS and therefor a showstopper for a planned production cluster. Looking forward to when this is shipped.:

Some additional work remains on the etcd operator side to support the self-hosted etcd case.

I'll close this and open a new tracking issue for self-hosted etcd (as that is primarily going to be an external dependency).

Opened https://github.com/kubernetes-incubator/bootkube/issues/445 to track self-hosted etcd + tls


Python 3.8 pip install error: "cannot import name 'CancelledError'"

Checklist

Python version 3.8
Does issue exists against the master branch of aredis? Yes

Steps to reproduce
pip install aredis
Expected behavior
It installs
Actual behavior
Collecting aredis
  Using cached https://files.pythonhosted.org/packages/26/a6/6df2c18fd3fcefa4ca978679bd280b610e686669efa2a0df53cc172a49a6/aredis-1.1.5.tar.gz
    ERROR: Command errored out with exit status 1:
     command: 'c:\program files\python38\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Simon\\AppData\\Local\\Temp\\pip-install-qmzu_4f_\\aredis\\setup.py'"'"'; __file__='"'"'C:\\Users\\Simon\\AppData\\Local\\Temp\\pip-install-qmzu_4f_\\aredis\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\Simon\AppData\Local\Temp\pip-install-qmzu_4f_\aredis\pip-egg-info'
         cwd: C:\Users\Simon\AppData\Local\Temp\pip-install-qmzu_4f_\aredis\
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Simon\AppData\Local\Temp\pip-install-qmzu_4f_\aredis\setup.py", line 6, in <module>
        from aredis import __version__
      File "C:\Users\Simon\AppData\Local\Temp\pip-install-qmzu_4f_\aredis\aredis\__init__.py", line 1, in <module>
        from aredis.client import (StrictRedis, StrictRedisCluster)
      File "C:\Users\Simon\AppData\Local\Temp\pip-install-qmzu_4f_\aredis\aredis\client.py", line 3, in <module>
        from asyncio.futures import CancelledError
    ImportError: cannot import name 'CancelledError' from 'asyncio.futures' (c:\program files\python38\lib\asyncio\futures.py)
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

@psidex This bug is fixed in issue #121
Doesn't that change take effect?

The latest version on pypi is 1.1.5 which does not contain that fix.
Is it possible for you to push the update onto there?

Sure, new version will be released this week

@psidex release 1.1.6 is public, i will leave this issue open (there may be some other implicit problems i am afraid of) Please try and if there is any problem, let me know and i will fix it as soon as possible~

Awesome, thanks 👍

FYI I was able to install the module on python 3.8.0 on my mac.
(venv) cobra$ pip install aredis
Collecting aredis
  Downloading https://files.pythonhosted.org/packages/54/6e/086022ccea78fd587303edb271c0c9100a7f65500e83b07bd69a44a4e65d/aredis-1.1.7.tar.gz (128kB)
     |████████████████████████████████| 133kB 10.6MB/s
Building wheels for collected packages: aredis
  Building wheel for aredis (setup.py) ... done
  Created wheel for aredis: filename=aredis-1.1.7-cp38-cp38-macosx_10_9_x86_64.whl size=90666 sha256=9c38a2d31d3c6475e6cd5a64aa8cbaca6d1641cf88dc99342b546a9808518dee
  Stored in directory: /Users/bsergeant/Library/Caches/pip/wheels/f4/4d/37/5d6f066872f806e09a844d60c231614ff68e0cc792ee90647e
Successfully built aredis
Installing collected packages: aredis
Successfully installed aredis-1.1.7
(venv) cobra$ pip --version
pip 19.3.1 from /Users/bsergeant/src/foss/cobra/venv/lib/python3.8/site-packages/pip (python 3.8)


Update requirements.txt

I think I'm going to move to depending on Altair, in which case we won't have to directly reference vega/vega3 here.

Excellent. Feel free to close this PR then.

Should we close this since pdvega now uses altair?

Yes, we need to update to depend on Altair instead.


tvm.tensor.HybridOp doesn't have reduce_axis

Is hyrbid script supporting the schedule of its reduce_axis?

Please open a new thread on the discuss forum https://discuss.tvm.ai/ I guess the current answer to the question is no. However, we can improve the new ir to support more such features in hybrid script


[Question]: I'm unable to use the & operator in testFiltercriteria in VSTest@2

Task name
No response
Task version
No response
Environment type (Please select at least one enviroment where you face this issue)

[X] Self-Hosted
[ ] Microsoft Hosted
[ ] VMSS Pool
[ ] Container

Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
Windows 10
Question
Hi, I am using VSTest@2 in my pipeline https://learn.microsoft.com/en-gb/azure/devops/pipelines/tasks/reference/vstest-v2?view=azure-pipelines&viewFallbackFrom=azure-devops
we are able to use the 'testFilterCriteria' like so in the yaml
testFiltercriteria: 'Category=SomeCategory'
however, when following the guidance on Expression/Operators, we are unable to use & | , the project doesn't detect any tests whatsoever.
https://github.com/Microsoft/vstest-docs/blob/main/docs/filter.md
e.g.
testFiltercrtieria: 'Category=Cat1 & Category=cat2'
have tried with paranthesis '(Category=Cat1) & (Category=cat2)' nothing is working.
tags are definitely correct.

any help/support would be appreciated. thanks

This works fine when I follow the syntax/language VsTestConsole and run locally on the .dll

JosephConroy93 Try without spaces:
testFiltercrtieria: 'Category=Cat1&Category=cat2'

@JosephConroy93 Have you been able to solve this ?
I'm running into the exact same issue.
It seems that it only evaluates the first part of an expression in the pipeline, and there's no way (AFAIK) to enclose multiple expressions in double quotes, which would make it work since I can replicate the behavior through the console locally.
Example
# This works
VSTest.Console /dll/path --TestCaseFilter:TestCategory=Category1

# This only runs tests with category: Category1
VSTest.Console /dll/path --TestCaseFilter:TestCategory=Category1|TestCategory=Category2

# This runs tests with Categories 1 & 2 (Note the double quotes)
VSTest.Console /dll/path --TestCaseFilter:"TestCategory=Category1|TestCategory=Category2"

So it seems that a solution for this would be to have the ability to enclose the expressions in the pipeline's task as well

Hi @shtlrs - sorry for the late reply, no I still don't have a solution to this unfortunately (which I find quite shocking!) - I can run with the '&' operator in the local VsTestConsole.exe runner. (as per the screenshot previously in the thread)
As for the pipeline, it doesn't work.

This worked for me
testFiltercriteria: 'Category=cat1 | Category=cat2 | Category=cat3'

Just noting the misspelling of testFiltercriteria here, if that's really what's in the yaml it could definitely cause it not to work.


Implement listen address argument and Rapid port argument

Issue #, if available: None
Description of changes:
Implements listen address argument (-l|--listen <listen-address>). If the argument is not supplied, it defaults to <IP_ADDRESS>:8080 for backwards-compatibility.
Not sure whether to merge on develop or main, but develop is the default branch so I'm basing it on that.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This would also close #14, which is a WIP with the same goal, but is based on a much older version of the codebase.

The motivation behind the PR is that I want to run multiple instances of RIE, to support multiple Lambda functions (without Docker, as I want the Lambda functions to quickly restart). However, while testing this with the current codebase, I noticed that port 9001 is also hard coded.
This seems to be for the rapid server. I'm not too familiar with the codebase, but I don't really see why there's another HTTP server. Could you give some information on this?
If it is essential to run those two servers, I propose adding another property to the SandboxBuilder Init object, that specifies the rapid server address & port, which in turn can be specified by an argument in the aws-lambda-rie package. Does that sound alright?

Changed the implementation up a bit from what I said earlier, but accomplishes the same goal for the Rapid port.

Is there any chance that this PR will be merged to the main branch and released?

This would be a very welcome feature, having a flag for the port would make using RIE much more pleasant in my workflow.

Thank you StefanNienhuis for working upon this improvement and opening a pull request.
We have merged functionality upstream to enable the providing of --runtime-api-address and --runtime-interface-emulator-address as command line arguments.
This functionality will be available with the next merge from upstream. I will leave this ticket open until the next merge from upstream.
Thanks you!

@bodelsc Could you provide the upstream package and commit this change was merged with?

Hi @StefanNienhuis . I'm sorry for the really long delay on our side. The changes mentioned previously were finally merged from upstream and released in the latest version.
Now it's possible to add a parameter --runtime-interface-emulator-address to indicate the address where RIE will be receiving requests.


Editor: Provide Inspector property settings for size Sprite with pixel size

Godot version:
All
** Issue **
Provide Inspector property settings for size Sprite with pixel size in addition to scale.

This does not appear to be a good idea, see discussion in #27935 and closing comment at https://github.com/godotengine/godot/pull/27935#issuecomment-501258590.


Fix UseCompatibleCmdlets rule

Add cmdlets exported by Microsoft.PowerShell.Core snapin
Ignore aliases
Ignore custom defined functions
Update command data files to PSv6.0.0-alpha.11

Resolves #639

This change is

Reviewed 9 of 9 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Comments from Reviewable


How can i add footer for each section?

I want to create footers for each section . How do it using your library. I m able to create section header.
Thankyou.

Hi, @developerShubhamAgr. This library doesn't support footers.

It would be great if you could add footers too. or Can you please recommend some other library with headers and footers based on category.
Thankyou.

Currently I'm not planning to add footers in the nearest future. Sorry for that.
You can check out https://github.com/afollestad/sectioned-recyclerview, but It's deprecated now.


Are you planning to support f2-nerf?

Thanks for your amazing work, are you planning to support f2-nerf, which shows great improvement on free cam trajactories

+1

Compared to Nerfacto, I achieved better results using F2-Nerf.
Nerfacto

F2-Nerf.

F2-nerf achieved clearer results even in object-centric scene, but with lose of finer objects.
Nerfacto

F2-Nerf.

It is not something we are actively working on. If someone else picks it up, we would love to register it with nerfstudio, instructions here.

Compared to Nerfacto, I achieved better results using F2-Nerf.
Nerfacto
F2-Nerf.

@xiemeilong is the source code of your implementation of F2-NeRF  within NeRF-Studio available?

@xiemeilong I'm using the official code of F2-NeRF.


Update enrollment codes to enums

It looks like the enrollment codes are strings, rather than numbers (enums would be OK).
It also looks like enrollment code was set to null on an error (rather than an error code). Here's my suggestion:
0 - participant was included in experiment
1 - participant excluded due to unspecified error
2 - participant reached experiment point prior to experiment enrolling
3 - participant was on exclusion list
4 - participant’s group was on exclusion list
5 - participant excluded due to group assignment logic
6 - participant excluded from group-assignment experiment - no group specified
(we should try not to use CarnegieLearningWeb/educational-experiment-service#1 - any "common" errors should get their own code).

@SritterCL we had some more discussion on this. The first step towards solving this is to finalize our first list of codes, then publish those as enums in the upgrade_types NPM package, and then go on to change the backend code.
Here is your last list of codes where I split out the INCLUDE into ASSIGN and ENROLL:

ASSIGN - participant was assigned a condition in the experiment
ENROLL - participant was enrolled in the experiment (as a result of the mark() call)
ERROR - participant excluded due to unspecified error
REACHED_PRIOR - participant reached experiment point prior to experiment enrolling
PARTICIPANT_ON_EXCLUSION_LIST - participant was on the exclusion list
GROUP_ON_EXCLUSION_LIST - participant’s group was on the exclusion list
EXCLUDED_DUE_TO_GROUP - participant excluded due to group assignment logic
NO_GROUP_SPECIFIED - participant excluded from the group-assignment experiment - no group specified

We don't know when we will assign the ERROR as the enrollment code, but we can publish it as an enum for now and find out the logic for it later. Let us know if you think the list is good to go.
@msandbothe if we change the upgrade_types NPM package and merge the PR, will it get published automatically in the public NPM repository?

7\. **EXCLUDED_DUE_TO_GROUP** - participant excluded due to group assignment logic

Can we have EXCLUDED_DUE_TO_GROUP_LOGIC

@jerith666 Do you know the answer to the publishing changes to upgrade_types NPM package? (@msandbothe if we change the upgrade_types NPM package and merge the PR, will it get published automatically in the public NPM repository?)

I don't; @mfugate1 should be able to tell us.
I thought everything was here in one repo, though, so I'm curious why it's necessary to wait for an artifact to be published at all.

@SritterCL We had a discussion on this issue while working on it and we have decided to include one more enum called INCORRECT_GROUP_OR_WORKING_GROUP = 'participant's group or working group is incorrect'.

OK - In general, it's fine to add new codes. Ideally, ERROR should never come up, since everything's covered under a different code. I assume "group of working group is incorrect" means that the group/WG was specified but isn't a valid group.

Yes, you are right. It means that the group/WG was specified but isn't a valid group. I will update the enum with the new suggested name.

@ppratikcr7 status update on this issue pls?

@amurphy-cl Majority of the changes were done. We will pick this up again.

During the implementation of #342 we have added enrollment and exclusion tables and the above code will be split into two different type ENROLLMENT_CODE and EXCLUSION_CODE
export enum ENROLLMENT_CODE {
  ALGORITHMIC = 'participant enrolled via algorithm',
  GROUP_LOGIC = 'participant enrolled due to group enrollment',
}

export enum EXCLUSION_CODE {
  ERROR = 'participant excluded due to unspecified error',
  REACHED_PRIOR = 'participant reached experiment prior to experiment enrolling',
  REACHED_AFTER = 'participant reached experiment during enrollment complete',
  PARTICIPANT_ON_EXCLUSION_LIST = 'participant was on the exclusion list',
  GROUP_ON_EXCLUSION_LIST = 'participant’s group was on the exclusion list',
  EXCLUDED_DUE_TO_GROUP_LOGIC = 'participant excluded due to group assignment logic',
  NO_GROUP_SPECIFIED = 'participant excluded due to incomplete group information',
  INVALID_GROUP_OR_WORKING_GROUP = "participant's group or working group is incorrect",
}

ASSIGN and ENROLL codes are removed as now we are not storing the assignments but only the enrollments. Let us know if any changes are required?

@VivekFitkariwala this looks good, I'll close this now.

Reopening because the code is in PR.


Fixed coveralls not found on MacOS with py39-311

Was reviewed in https://github.com/zhmcclient/python-zhmcclient/pull/1664 - no separate review needed.

Pull Request Test Coverage Report for Build test-54
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 22.535%

Totals

Change from base Build test-49:
0.0%

Covered Lines:
128

Relevant Lines:
568

💛  - Coveralls


Can I specify one URL and use PHP to retrieve only one bluesky post by someone else?

I want to retrieve someone else's specific post. if I am using Bluesky and happen to find a post I like, can I use PHP to retrieve that one post? I want to embed it in my blog and display it.

You might want to use a more generic oEmbed PHP library like https://github.com/ricardofiorani/oembed which can support any oEmbed endpoint, including Bluesky.
https://embed.bsky.app/oembed?url=https://bsky.app/profile/mkarolian.bsky.social/post/3lbunm54agc2k

Sorry @e1blue, but this is probably beyond the scope of the library...


You need to be more careful with Migrations

Description
Please take a look at the following migration: Shopware\Core\Migration\Migration1583142266FixDefaultOrderConfirmationMailTemplateVATDisplay
https://github.com/shopware/platform/blob/master/src/Core/Migration/Migration1583142266FixDefaultOrderConfirmationMailTemplateVATDisplay.php
The migration fetches the system default mail template of a given type (in this case the order confirmation mail) and updates / overwrites the mail template translations to fix a given problem.
This is problematic. You allow your users (system administrations) to edit their mail templates. The system does not indicate in any form, that system default mail templates can be overwritten by migrations in the future. The system also does not prevent the user to edit system default mails, like it is done with system default cms page layouts.
So what's happening is, is  that users are editing their templates (and everyone who has ever worked on an email template knows how painful this experience can be), and these templates have now be overwritten by a migration.
Even worse: This is a change, many users won't even recognize. The change of an order confirmation email, for example, is not something system administrators regularly check. It's possible, that in many live systems the email template of the order confirmation templates is now broken (and I mean aesthetically, not syntactically), and the shop administrators don't even know about it.
That's not great.

Hi @AlexBachmann,
we are aware of this. We are discussing methods to test our migrations better and are setting up migration rules as we speak. We are very sorry for any inconvenience we caused.
If you have some suggestions, I'm all ears :)
Sunny regards
Pantrtxp

I love the migration system you. So I guess, the only clean solution is to make system default email templates not edible. A user who wants to change it's templates then needs to copy the default template, change it, and assign it to it's sales-channel.
If you're really good, you create some kind of "wizzard" arround it, to help with the workflow.

The Wizard would be lovely but sadly there are more important issues right now.
We are going to disable the ability to edit system default
You can track the state of this here
Sunny regards
Pantrtxp

I am closing this Issue due to inactivity.
You can track the state of our progress with the given link at any time.
@AlexBachmann, thank you for your concern :blue_heart:
Sunny regards
Pantrtxp


UPGRADE.md file missing with broken link for php-cs-fixer self-update

Bug report
From PHP CS Fixer updated (2.18.2 -> v3.0.2)
$ php-cs-fixer self-update
A new major version of PHP CS Fixer is available (v3.0.2)
Before upgrading please read https://github.com/FriendsOfPHP/PHP-CS-Fixer/blob/v3.0.2/UPGRADE.md
If you are ready to upgrade run this command with -f
Checking for new minor/patch version...

The file https://github.com/FriendsOfPHP/PHP-CS-Fixer/blob/v3.0.2/UPGRADE.md is missing

thanks for reporting the issue!
I addressed it in https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/5881 , but it will be released as 2.19.2, so if you use 2.19.1 or lower, it will still have the bug.
Feel free to propose a PR that would suggest to update to latest MINOR of current MAJOR before bumping the MAJOR


Remove quotes around env variable name in resolveBuildSecretEnv

When trying to implement an input for secrets defined by environment variables to the build-push-action I made use of the new resolveBuildSecretEnv implemented in: https://github.com/docker/actions-toolkit/commit/a1ffbe96065f5934066c01459a17967107273cc6.
I implemented a test for the new input and it didn't work due to csv parsing breaking because of the quotes around the env variable name as can be seen here. After monkeypatching this function (removing the " around the env variable name) and building again it works correctly as can be seen here.
Pasting images since action runs disappear after a while:

@crazy-max sorry for the ping, I hope this can get merged as it is a quite blocking issue for reading env variables to secrets. After this I will create a pull request in the action to make use of this change.

LGTM thanks! And sorry for the delay!
About the implementation on build push action, see my comment #110 (comment)

No worries!
Just like he said i do not have access to providing inputs directly to the action. In my case it lives in a reusable workflow which can define environment variables but not like that directly. In our case environment variables can be defined dynamically when running the workflow including env vars that contain secrets. There is also another input to this workflow that says what secrets should point to what environment variables.

@crazy-max just a few more questions:
When will a new npm package version of this be released?
In the pull request to the GitHub action should i bump the package version with it?
❤

Released https://github.com/docker/actions-toolkit/releases/tag/v0.13.0-rc.1 if you want to bump


Stuck when mup deploy

I am having an issue where when i run mup setup, everything goes fine, but then I run mup deploy and it never finishes. It says "Building App Bundle Locally" and gets stuck. I have left it for over 24 hours and it never finishes or errors out. Here is my code, I am trying to use abernix/meteord:base as a docker image, however I have also tried with out specifying an image at all.
Here is my mup.js file
`module.exports = {
  servers: {
    one: {
      host: 'xxx.xxx.xxx.xx',
      username: 'xxx',
      // pem:
      password: 'xxxxx'
      // or leave blank for authenticate from ssh-agent
    }
  },

  meteor: {
    name: 'app',
    path: '.',
    servers: {
      one: {}
    },
    buildOptions: {
      serverOnly: true,
    },
    env: {
      ROOT_URL: 'https://xxx.xxx.xxx.xx',
      MONGO_URL: 'mongodb://xxx:xxx@aws-us-east-1-portal.19.dblayer.com:10960,aws-us-east-1-portal.20.dblayer.com:10812/admin'
    },
    ssl: {
      crt: '.', // this is a bundle of certificates
      key: '.', // this is the private key of the certificate
      port: 443 // 443 is the default value and it's the standard HTTPS port
    },

    dockerImage: "abernix/meteord:base",
    deployCheckWaitTime: 100
  }
};`

when I run mup logs --tail=50, all that returns is
[xxx.xxx.xxx.xx] Error: No such container: app
I am running Windows 10 on my machine, and the servers I have tried to push to are running Ubuntu 14.04.5 at digital ocean.
Thank you, I appreciate the help.

It appears that this problem was unrelated to Mup.
I was having an issue where Meteor was trying to install<EMAIL_ADDRESS>but would never finish the installation. Therefor when mup tried to build it locally, it never could.
Anyway I fixed the problem by updating Git, following the advice given by NenadPavlov under this post - https://github.com/meteor/meteor/issues/7688.


Introduce common preset for Jest test configuration

Description

We are repeating a lot of configuration for our tooling. There is a way to create presets for Jest and other tools. This is a first try that should present the preset for Jest, that can be used by all packages.
There will be some future steps that will introduce usage of setup files and general mocking. This PR is now moving most common configurations into the preset. It tries not to break everything.
There will be also move of the jest.config file from the config directory to the root of each package. Please consider if this is a good approach, but but it seems to me, that all packages across the JS-verse is now adopting the similar naming of the configuration file and that is *.config.{js, ts, mjs, cjs}. Actually I think it is a good move but I am missing now the files starting with the dot ., becuase they are naturally sorted by name and I now that all the file with the starting dot are configurations. Now with the jest.config and eslint.config and prettier.config it will be really messy there. 🤷♂
Additional context

https://github.com/carbon-design-system/carbon/blob/e4922a44b2110b9dcb71da21ff6c0c9bb95610e0/config/jest-config-carbon/index.js

Issue reference

coverage: 80.657%. remained the same
when pulling 8615c83a85d9cd9a9eb0b71c599ac321815921b7 on chore/jest-config-spirit
into 71df455a3f997f9647877ae3db572db11e4c66f0 on main.


Invalid link to Discord in the README

What happened?
https://github.com/cookiecutter/cookiecutter-django/commit/5ef345bd3a5c180db29d5c52b60b66d042251488 has updated the link from Slack to Discord in the README.
But the new Discord link results in "Invite Invalid".
What should've happened instead?
The Discord server for Cookiecutter-Django should have been accessed.
Additional details
No additional details should de necessary beyond clicking on the link. Unfortunately, I don't know the correct Discord link to PR.

This one works? https://discord.gg/9BrxzPKuEW

Yes, it does.


License choice checklist

Setting up this issue for discussion of if we should change our licensing in our Python and R issue. Feel free to leave your thoughts and discuss.

Having taken a look at the lecture 8 material and as discussed with @lipcai  and @scarlqq on the call yesterday, we are going to stick with the MIT license.
Since our package seems more for our own purposes of software development and workflows conceptual understanding MIT seems to fit what we require according to https://choosealicense.com. People can do what they want with it, but we have limitation of liability.
To opt MIT license for our package , we have added contributor's name already.

We all agree with this. Considering the purpose of our package, MIT license should be exactly what we need.

We all agree with this. Considering the purpose of our package, MIT license should be exactly what we need.

Thanks for the discussion @lipcai @scarlqq. Looking at your comments to go with existing MIT license, I'm closing this issue


[Error] java.lang.IllegalStateException: onComplete must not be called after onError

I am using mqtt-bench ( https://github.com/takanorig/mqtt-bench )
When I benchmark with "./mqtt-bench -broker=tcp://<IP_ADDRESS>:1883 -action=pub -count=5000" there are some errors:
[Error] java.lang.IllegalStateException: onComplete must not be called after onError akka://jetmq/user/<IP_ADDRESS>:56796 class akka.actor.OneForOneStrategy onComplete must not be called after onError

@spidercpsf Thanks a lot!
Missed this thing, now fixed:
https://github.com/butaji/JetMQ/blob/a366a5959927d3f864d0b51dd57bbedba73906b9/core/src/main/scala/net/jetmq/broker/TcpConnectionActor.scala#L34
BTW Do you have some success benchmarking MQTT brokers?  I'm really interested to do something like that

Yes, I am taking some benchmarks with MQTT brokers, and trying to find the way to optimize mqtt with akka and scala.


is colyseus suitable for turn base game?

Hi!
I am develop a turn base game (likes tic tac toe online). In game, users start match and play turn by turn, user move is limited by time (e.x 30 second each move) . And after some conditional, match over and then user start match again . So how can i implement this feature in colyseus server?
Thank so much!

Hi @takaaptech,
Yes, it's possible.
I'm starting to write the documentation for the server, you can take a look here: http://gamestd.io/colyseus/docs/room-state.html (feedbacks are welcome!)
Considering you want to make a "tic tac toe" like game, you'd need to allow only 2 clients per room, and the room state might look like this:
var Room = require('colyseus').Room
class TicTacToeRoom extends Room {
  constructor (options) {
    super(options, 1000)

    this.players = {}
    this.setState({
      currentPlayer: null,
      board: [[0,0,0], [0,0,0], [0,0,0]]
    })
  }

  requestJoin() {
    return this.clients.length < 2
  }

  onJoin(client) {
    this.players[client.id] = (this.clients.length == 1) ? "X" : "O"
  }

  onMessage (client, data) {
    // change this.state.board[x][y] to client's attribute (X or O?)
    this.state.board[x][y] = this.players[client.id]
  }
}

module.exports = TicTacToeRoom

I didn't tested this code, just wrote it quickly here. You'd basically have to deal with the board state in the server, and request commands through the client on the position you want to update. When the state has successfully changed, you can apply the change in the client-side through the patch callback in the client.
To deal with timing events, you could use built-in setInterval and setTimeout methods, and switch the state.currentPlayer variable. It's also a good idea to check for the currentPlayer when accepting movements inside onMessage method.
Hope this helps, let me know if you have any questions.

Thank so much!


Chore/danger me

自分自身のPRでプラグインを実行する

Messages

:book:
PR Title: Chore/danger me

  Generated by :no_entry_sign: dangerJS against 0b08a198278789a2c848e7ee80c702a2026f931c

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Sprite Frames preview off by 1 pixel

Godot version
v4.0.2.stable.official [7a0977ce2]
System information
Windows 11
Issue description
Sprite Frames preview is off by one pixel, even though playing the animation in-game or in the 2D view doesn't present the issue that should be visible from the Sprite Frames .tres preview.

Steps to reproduce

Obtain a sprite sheet of DIM1xDIM2 dimensions, where at least one cell has pixels on the border with a neighbor cell below it.
Open up the sprite in the Sprite Frames editor, and input DIM1 and DIM2 in the dimensions at the top.
Try to create an animation (from a single row of this sheet), which contains the mandatory cell described above.
Try to create an animation using cells from a row below the one from (2.). Ensure the cell below the special cell described at (1.) is also selected.
Look at the preview for these two animations. It looks like a single pixel row that should be present on frames from (2.) are present on (3.) instead.
Observe that using these two animations in-game or in the 2D preview doesn't present this issue.

Minimal reproduction project
for-bug-report.zip

I realize now that this is actually a side-effect of the issue from here: https://github.com/godotengine/godot/issues/74180 . If I select the linear filter (instead of the Nearest filter, since I was dealing with pixel art), then this issue does indeed occur. I'm closing it as a duplicate of that ticket (with the same fix from that issue: https://github.com/godotengine/godot/pull/73962 ).


how to generate types.generated.go

could u support a script for generate *.generated.go in this project?

The same way k8s produces its types.generated.go.
go get github.com/ugorji/go/codec/codecgen
codecgen -d 1234 -o types.generated.go types.go

And for zz_generated.defaults.go. You will need to build kubernetes/cmd/libs/go2idl/defaulter-gen.
And run in /pkg/cluster
defaulter-gen -i .

Yes. We should probably have a script to do this and maybe a guide for hackers somewhere in the repo.


fix: repository providers, handling of private/public repos, added un…

…ittests

tests do not pass


Changing BrandImagery to Logo and moving it from Patterns to Elements

Fixes
Fixes issue #135 by @annatuma, @hugosolar and @dhruvkb
Description

This changes the folder and filenames from BrandImagery to Logo.

This also changes the location of the new Logo folder from Patterns to Elements.

Type of PR
This PR is a feature.
Checklist:

[x] My pull request has a descriptive title (not a vague title like "Update index.md").
[x] My pull request targets the master branch of the repository.
[x] My commit messages follow [best practices][best_practices].
[x] My code follows the established code style of the repository.
[ ] I added tests for the changes I made (if applicable).
[x] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible errors.

Legal
By submitting this PR, I agree to abide by the terms of the Developer
Certificate of Origin.

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Bob160 I see you've added a lot of SVGs in the codebase. That's not the right thing to do because the assets need to be used directly from the @creativecommons/vocabulary dependency.

So when you want to access the logo you need to use the path
@creativecommons/vocabulary/assets/<path_to_svg>

Closing as the issue has itself been closed.


Update all pubspecs to work with the latest version of pub and Flutter

Description

Migrated all plugins to the new [plugin platforms manifest] (https://flutter.dev/docs/development/packages-and-plugins/developing-packages#plugin-platforms)
Removed author: and authors: fields from all pubspecs as the fields are deprecated, moved author names from various plugins to the repository's AUTHORS file.

Presubmits are timing out 😞
I'm landing this PR and going to monitor post-submits before publishing any of the plugins.
cc @collinjackson


[VS Code] The user code is invalid or expired. Try signing in again.

Error:
The user code is invalid or expired. Try signing in again.
Steps to Reproduce:

Version Data

extensionName
VSLS

extensionVersion
0.3.472

protocolVersion
2.2

applicationName
VSCode

applicationVersion
1.25.1

platformName
Linux

platformVersion
4.13.0-45-generic

To help us investigate it, can you please get Live Share logs from VS Code and attach them to the issue? To get the logs, run 'Live Share: Export Logs' command from VS Code command palette. The command will ask you for the zip file location, and will save the logs into that file. You can then attach the file to the issue.

@bkamapantula The two valid cases where this error happens are:

User tries to use the user code after 2 minutes the user code was issued.
User tries to use the user code twice(it is valid only 1 time).

Do you think either of those could have happened in your case?

Thanks @avanderhoorn, @srivatsn. 2) happened in my case
After few tries I was able to setup the Live Share the same day. Currently it runs fine (attaching the logs regardless).
vscode-liveshare.zip
marking this as addressed, closing it

Thanks for getting back to us!


increased opacity on muted channels for bot-icons and avatars (with a…

Summary
increased opacity on muted channels for bot-icons and avatars (with and without status icon)
Ticket Link
MM-36357
Screenshots

Before
After

EDIT
changed before image, since it was not in muted state
NONE

@michelengelen I think the title for this PR should be 'reduce opacity…' rather than 'increase opacity…'?

@michelengelen I think the title for this PR should be 'reduce opacity…' rather than 'increase opacity…'?

My bad! All because I cannot get my head straight today! 🙃


Spectrum m4i improvements

Added asynchronous acquisition with setup_multi_recording(), start_triggered() and get_data()
Add 20 MHz low-pass filter
Added firmware to get_idn()
Improved error logging
Use f-strings
Improved documentation

@sldesnoo-Delft
Thanks for the PR. I will review it as soon as possible.


@testing-library/jest-dom and swr do not play nicely together

Bug report
Description / Observed Behavior
I'm refactoring an application to use swr data fetching hooks. The repo uses jest for testing, and specifically uses @testing-library/jest-dom for better test matchers. Keeping @testing-library/jest-dom and importing swr in a source file and/or a test file throws an Illegal invocation TypeError:
TypeError: Illegal invocation

    >  8 | import { cache } from 'swr'
         | ^

      at addEventListener (node_modules/jsdom/lib/jsdom/living/generated/EventTarget.js:18:13)
      at Object.initFocus (node_modules/swr/dist/index.js:1:1981)
      at P (node_modules/swr/dist/index.js:1:3845)
      at Object.<anonymous> (node_modules/swr/dist/index.js:1:3915)
      at Object.<anonymous> (importSwr.test.js:8:1)

Commenting out the @testing-library/jest-dom import fixes the issue.
Expected Behavior
I'd like to be able to test a component that uses swr for data fetching, and use extended matchers from @testing-library/jest-dom for assertions.
Repro Steps / Code Example
I made a minimal repro branch in my larger open source codebase. For your convenience:
git clone git@github.com:glifio/wallet.git (or https://github.com/glifio/wallet.git)
npm i
git fetch origin debug/jest-swr-import
git checkout debug/jest-swr-import
npm run test:watch importSwr

The files to look at are in this diff. Importing anything from swr in the test file itself breaks the tests. Here's the @testing-library/jest-dom import.
Additional Context
"swr": "^1.0.0",
"@testing-library/jest-dom": "5.5.0",
"@testing-library/react": "10.0.1",
"babel-jest": "26.6.3",
"jest": "^25.1.0",

Thanks for your reporting.  However this problem might not be related to SWR itself.
I tried to reproduce you problem and i did see the error. But commenting out the @testing-library/jest-dom will not fix the issue.
SWR also uses @testing-library/jest-dom for testing.
https://github.com/vercel/swr/blob/c00f0b47a01a2e89e027df6d4ec6a2986195d50c/jest-setup.ts#L1

@Schwartz10 I'm using Enzyme + Jest and ran into a similar issue (Illegal invocation at addEventListener in EventTarget.js whenever swr was imported). fwiw I solved it by upgrading jsdom from 14 to 16.


Request for Pre-trained Checkpoint Release

Dear authors,
Thank you for the excellent work you've done with the CheX project and congratulations on the acceptance of your paper to ECCV 2024! I am currently working on a project that could greatly benefit from the use of your pre-trained model. Could you kindly provide an update on when we might expect the checkpoint to be available?
Thank you for your time and effort in making this resource available to the community.
Best regards

We have now made the checkpoint of ChEX available.
Please follow the instructions in the README to download and use the checkpoint.
If you encounter any problems with the checkpoint, feel free to open a new issue.


connection issue

I would say that you have done excellent work, however I am facing the problem with the script :-
_Setup for bulb started ...
Step 1: Read characteristics from bulb 1 ...
connect error: Connection refused (111)
while executing
"exec gatttool -b $mac --characteristics > $hndfile"
(procedure "setupCharacteristics" line 5)
invoked from within
"setupCharacteristics"
(procedure "setup" line 13)
invoked from within
"setup"
("^setup$" arm line 2)
invoked from within
"switch -regexp $c {
^status$ {
init [expr $INIT(color) + $INIT(effect) + $INIT(timers)]
printStatus
}
^dump$ {
init [exp..."
(procedure "doCommand" line 10)
invoked from within
"doCommand"
("while" body line 2)
invoked from within
"while {$cmd != ""} {
doCommand
}"
(file "./mipow.exp" line 1641)_
however when I run the command directly I get the output
gatttool -b 90:F6:4B:10:AC:E6  --characteristics
handle = 0x0002, char properties = 0x0a, char value handle = 0x0003, uuid = 00002a00-0000-1000-8000-00805f9b34fb
handle = 0x0004, char properties = 0x02, char value handle = 0x0005, uuid = 00002a01-0000-1000-8000-00805f9b34fb
handle = 0x0006, char properties = 0x02, char value handle = 0x0007, uuid = 00002a04-0000-1000-8000-00805f9b34fb
handle = 0x0009, char properties = 0x22, char value handle = 0x000a, uuid = 00002a05-0000-1000-8000-00805f9b34fb
handle = 0x000d, char properties = 0x10, char value handle = 0x000e, uuid = 00002a37-0000-1000-8000-00805f9b34fb
handle = 0x0010, char properties = 0x0e, char value handle = 0x0011, uuid = 00002a39-0000-1000-8000-00805f9b34fb
handle = 0x0012, char properties = 0x02, char value handle = 0x0013, uuid = 0000fff8-0000-1000-8000-00805f9b34fb
handle = 0x0014, char properties = 0x0a, char value handle = 0x0015, uuid = 0000fff9-0000-1000-8000-00805f9b34fb
handle = 0x0016, char properties = 0x06, char value handle = 0x0017, uuid = 0000fffa-0000-1000-8000-00805f9b34fb
handle = 0x0018, char properties = 0x06, char value handle = 0x0019, uuid = 0000fffb-0000-1000-8000-00805f9b34fb
handle = 0x001a, char properties = 0x06, char value handle = 0x001b, uuid = 0000fffc-0000-1000-8000-00805f9b34fb
handle = 0x001c, char properties = 0x0a, char value handle = 0x001d, uuid = 0000fffd-0000-1000-8000-00805f9b34fb
handle = 0x001e, char properties = 0x0a, char value handle = 0x001f, uuid = 0000fffe-0000-1000-8000-00805f9b34fb
handle = 0x0020, char properties = 0x0a, char value handle = 0x0021, uuid = 0000ffff-0000-1000-8000-00805f9b34fb
handle = 0x0023, char properties = 0x12, char value handle = 0x0024, uuid = 00002a19-0000-1000-8000-00805f9b34fb
handle = 0x0027, char properties = 0x02, char value handle = 0x0028, uuid = 00002a25-0000-1000-8000-00805f9b34fb
handle = 0x0029, char properties = 0x02, char value handle = 0x002a, uuid = 00002a27-0000-1000-8000-00805f9b34fb
handle = 0x002b, char properties = 0x02, char value handle = 0x002c, uuid = 00002a26-0000-1000-8000-00805f9b34fb
handle = 0x002d, char properties = 0x02, char value handle = 0x002e, uuid = 00002a28-0000-1000-8000-00805f9b34fb
handle = 0x002f, char properties = 0x02, char value handle = 0x0030, uuid = 00002a29-0000-1000-8000-00805f9b34fb
handle = 0x0031, char properties = 0x02, char value handle = 0x0032, uuid = 00002a50-0000-1000-8000-00805f9b34fb
handle = 0x0034, char properties = 0x0a, char value handle = 0x0035, uuid = 00001013-d102-11e1-9b23-00025b00a5a5
handle = 0x0036, char properties = 0x08, char value handle = 0x0037, uuid = 00001018-d102-11e1-9b23-00025b00a5a5
handle = 0x0038, char properties = 0x12, char value handle = 0x0039, uuid = 00001014-d102-11e1-9b23-00025b00a5a5
handle = 0x003b, char properties = 0x02, char value handle = 0x003c, uuid = 00001011-d102-11e1-9b23-00025b00a5a5

Dear viks2015,
can you reproduce it all the time?
Maybe try to set the script into debug mode. Open the script mipow.exp and change the line no 33 from
log_user 0

to
log_user 1

Check the output and try to reproduce all commands manually.
Best regards,
Martin

I am getting this output now
Setup for bulb started ...
Step 1: Read characteristics from bulb 1 ...
connect error: Connection refused (111)
while executing
"exec gatttool -b $mac --characteristics > $hndfile"
(procedure "setupCharacteristics" line 5)
invoked from within
"setupCharacteristics"
(procedure "setup" line 13)
invoked from within
"setup"
("^setup$" arm line 2)
invoked from within
"switch -regexp $c {
^status$ {
init [expr $INIT(color) + $INIT(effect) + $INIT(timers)]
printStatus
}
^dump$ {
init [exp..."
(procedure "doCommand" line 10)
invoked from within
"doCommand"
("while" body line 2)
invoked from within
"while {$cmd != ""} {
doCommand
}"
(file "./mipow.exp" line 1641)

Ok, there seems to be a bug. I am investigating

I have uploaded a bugfix. Please try if this works for you.
Best regards

Works Now !!! Great Work. Can you port your code to pure python ?

How can I give percentage level of brightness instead of up and down option multiple times

Excellent.
Actually I have started porting the code to Python in order to use in in KODI. But the gatttool-wrapper is very instable. That is some code I have taken from here: http://flowcloud.github.io/ci20-bluetooth-LE/2015/09/10/bluetooth-control-in-python/
I don't expect to continue my work here because for my KODI plugins a use subprocesses instead.
Nevertheless, I have just uploaded the Python code that I have already. Note that it is not very stable as mentioned. See https://github.com/Heckie75/py-mipow-playbulb
Best regards,
Martin


No songs load

Describe the bug
When selecting "Start Game" from the main menu, the music selection screen shows no songs, even if you download new ones they don't show up.
To Reproduce
Steps to reproduce the behavior:
Unknown; this happened upon initially downloading the game, the bug persists when the software is restarted.
Expected behavior
Expected behaviour is to see the songs on the list on the right.
Screenshots

Desktop (please complete the following information):

OS: Windows 10 Home
Version: 10.0.18363 Build 18363
64bit or 32bit: 64bit

Additional context
Add any other context about the problem here.

go to main menu
options
select game
pick dance (you are on pump in this screenshot)
although it is a bug that this keeps happening, the game should not be defaulting to pump for new players


Memory allocation in C

Memory allocation using malloc_aligned. Also implements first_touch.

I have now addressed all the comments so far. This version is also passing all tests. I believe it should be good to go.

I'd really like to see at least a simple preliminary before-and-after comparison on a system with multiple NUMA domains before merging this. I just really want to be sure we are actually achieving what we think we're achieving and that we're not impacting the performance of our current benchmarks.

Since @FabioLuporini is leading the benchmarking cause, he is aware of a lot more "best-practices" and "recipes" than me. I was hoping it would be trivial for him to compare performance of this branch compared to master. Can you help, @FabioLuporini ?

Tests on bigger domains show that this entire approach fails for larger domains (which are when this approach is required).
More tests required before this is merged.

Experimentation is not always about just running tests, but also making sense of them :) so I prefer to instruct you on how to run tests, also because I spent a considerable amount of time on reproducibility.
First thing: I'm gonna merge #136 today, and any tests that you run should be rebased on the new master. This is because that branch fixes several performance measurement issues.
Secondly, to facilitate experimentation, you can temporarily cherry-pick on your branch this commit 4b0f4c7, which gives you a launcher.sh script that you can invoke (examples folder) to run various types of experiments. If you use this, it becomes easy to reproduce exactly the same experiment.
Now, to the PR.
Once we have some compelling evidence that we are dealing with NUMA correctly (runtime improvements aren't enough; we need some profiling, or perhaps -- simpler -- using a tool such as http://man7.org/linux/man-pages/man8/numastat.8.html), then we should write tests (showing that we aren't leaking memory, for example) and an example for performance reproducibility. The latter can be done by taking an example, and running it on a NUMA node in default mode and with first_touch somehow disabled.

Did I mention this PR cuts the memory footprint of the objects in half?

I can totally believe that, removing the deep copy looks like a very important fix. Well spotted!

TLDR: This branch doubles the Gflops based performance and reduces the memory footprint in half.
As of now this branch passes all tests on travis and all manual tests I ran here on larger domains (I have tested domains up to 500x500x500).
Without the NUMA related improvement, this branch already performs better than master in terms of GFLOPS/sec (28 Gflops on master vs 34 on this branch consistently across multiple runs). Master also currently allocates at least twice the required amount of memory because of the implementation of the python based alignment function and there is also an in-memory copy there which takes significant time for large domains. Removing all that, this branch reduces the memory footprint in less than half. The time saved in the in-memory copy is probably being offset by the additional overhead of more JIT operations but since a O(n) operation is being replaced by a O(1) one, we should see a total runtime improvement for larger domains.
Now, coming to the NUMA related improvements. Running on a single socket using numactl, we achieve 34 Gflops on a 300x300x300 domain on a node in Yemoja. Clear proof that first touch is happening would be if we can double the performance by running on both sockets. If we run it without numactl, the performance varies greatly between 40-70 - my suspicion was that this is due to threads migrating from one core to another. Rerunning it with KMP_AFFINITY set to pin each thread to a particular core explicitly confirmed this suspicion and we achieved 70 GFlops consistently - hence confirming that we are indeed using our NUMA channels optimally.
However, this performance was reached on a version of this code that was slightly modified from the version on this branch. This is because first touch initialization can not happen unless the initialization function has information about the compiler. Since the initialization function is being called from DenseData, our DenseData objects need a reference to the compiler object that will later be passed to the Propagator used to access it. It makes perfect sense since the compiler defines some aspects of how the Propagator will access the data structure and without knowing the access pattern, first touch initialization can not happen. In this case, it is the OpenMP flag but in the future we might be using more compiler specific initializations (e.g. if we start to set affinity internally). I am open to ideas on how to structure this in a way that is not considered "hackish".
Another thing is that these tests were without blocking. When the Propagator that will access the data has blocking enabled, the initialization function needs that information along with the block sizes that the Propagator will use. How to get that information inside the initialization function is another thing that I want to hear your "non-hackish" ideas on.

Allright, some questions:

When you say "Without the NUMA related improvement, this branch already performs better than master in terms of GFLOPS/sec (28 Gflops on master vs 34 on this branch consistently across multiple runs). " -- do you know why is that the case ?
as for "halving the memory requirements" -- do you mean that "Python" (u mean NumPy, right?) first allocates a memory region, and then to make it aligned, it performs a second copy? I'm not sure I understand, could you point me to some references?
feels like you're mixing thread pinning with NUMA-awarness. Obviously, thread pinning should always happen, even on a single socket. Those 70GFlops/s -- I assume you're on your branch. What if you run the same test, on a dual socket machine, clearly with thread pinning, but from master ? The achieved flop rate should be > 34 but much less than 70. In which case, we would have proof that something good is happening, but still not enough to claim "we have a perfectly working first touch implementation" (ie, we would have indirect evidence supporting our hypothesis, but probably we need to go deeper with profiling to be 100%).

The rest looks all very good
Thanks

@FabioLuporini Your first two points are related to the fact that we can't be sure (as far as I am aware) that numpy allocates memory aligned on page boundaries (or any such alignment). To work around that fact in python-land we (master) currently use this which is essentially a code snippet I found on stack overflow and it seemed to work decently at the time.
Now, as for your first point. Now that our memory allocation is using posix_memalign, it's probably aligned for the first time and something in our previous implementation was "tricking" us into believing that the memory allocation was aligned. This is my best explanation for this improvement that I see.
As for your second point, if you look at the same aligned function I linked to above (I am the author of that function - this is my fault to begin with), it expects a numpy array to be passed. It then allocates a second numpy array of the same size or slightly bigger, and then does a copy operation from the previous array to the new one - eventually returning a reference to this new "aligned" array. Very often, when I see a devito based benchmark run taking long and I pause the process using gdb to figure out what's taking so long - even before the actual execution of the JIT-ed code - I see the execution inside this copy operation. Given the nature of the way we use the aligned function here, this copy operation was probably unnecessary to begin with. However, I left it in there "for completeness" just in case someone decided to use it in a way that's not the same as how I used it in interfaces.py. I think this is the perfect opportunity to get rid of that function altogether. This is how we cut our memory footprint in half and remove this copy operation.

I'm happy about the second point, thanks, now it makes sense.
"I think this is the perfect opportunity to get rid of that function altogether" -> yes, please do it
as for point 1), NumPy has a flag to align arrays to cache boundaries, but not to page boundaries (I was looking for it to trigger linux' transparent huge pages, which may diminish TLB misses). But now we don't need this anymore.
point 3) is still pending :)
Thanks

On master:
[navjot.kukreja@service7 devito]$ KMP_AFFINITY="granularity=fine,proclist=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],explicit"  python examples/benchmark.py -P acoustic -d 300 300 300 -o
Applying Forward
IntelCompiler: Compiling /tmp/devito-1078/8e13b48868e8b6bb5cb052d76c13c9877168adaf.cpp
Shape: (320 x 320 x 320) - Cache Blocking: None
Time: 4.225890 s (1133 MCells/s)
Stencil: 1.687500 OI, 31.21 GFlops/s (time: 4.223773 s)

On this branch:
[navjot.kukreja@service7 devito]$ KMP_AFFINITY="granularity=fine,proclist=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],explicit"  python examples/benchmark.py -P acoustic -d 300 300 300 -o
IntelCompiler: Compiling /tmp/devito-1078/bcf80c70e8d713dcf18ce604381444f5249e081d.cpp
IntelCompiler: Compiling /tmp/devito-1078/7ea5b123bfc59e6b236b4e8d02143a482f53e940.cpp
IntelCompiler: Compiling /tmp/devito-1078/915211830732f5771c942910fb30e2a60406cc48.cpp
Applying Forward
IntelCompiler: Compiling /tmp/devito-1078/becf8f272f6129e85929cf25d453da17a945ee0d.cpp
IntelCompiler: Compiling /tmp/devito-1078/b8ff4d07ab6825a0059c84f2d8fb62bfec4004f2.cpp
IntelCompiler: Compiling /tmp/devito-1078/d2432c5ef46ba35b2a7cd547d348f0618103e6c3.cpp
IntelCompiler: Compiling /tmp/devito-1078/e20a64d85aefde306afdb1c9d658d4c2b1d8d629.cpp
IntelCompiler: Compiling /tmp/devito-1078/b6b8b5c89db0512de3fc12d7096d7a6f8ac392be.cpp
Shape: (320 x 320 x 320) - Cache Blocking: None
Time: 1.903779 s (2516 MCells/s)
Stencil: 1.687500 OI, 69.33 GFlops/s (time: 1.901353 s)

Impressive, great.
Just to be 100%: can we reproduce the same behaviour with tiling and autotuning on?

@FabioLuporini Read the discussion we have already had on this topic https://github.com/opesci/devito/pull/20

Are you asking me to do it because you expect the results to be different? I don't see what carrying out more tests will achieve other than just delaying this PR while others go in, making me spend more hours on rebasing again.
I think I have proven that this works by the tests done already. If you don't want to accept it, that's your decision.

I don't agree. I spent a considerable amount of time to achieve reproducible science in Devito; when it comes to performance it's best to be extremely cautious. Perhaps we should add performance regression tests.
So when you say "I don't see what carrying out more tests will achieve", I disagree. First touch should be orthogonal to cache blocking/autotuning, OK, but checking that it produces the expected performance results is a 5 minutes work. BTW, the PR isn't huge, so rebasing in case of more merges won't take hours. We can give priorities if this makes you more comfortable -- your PR will land first (out of those in the list), so you don't have to be worried about rebasing.
I think you've provided compelling evidence that this PR does many good things, but there's one last standing item.
In the end, I'm not asking you to produce plots or tables, but just to run a 5-min experiment (the one you already did with -a).

Rebasing across your last PR took me 4 hours.
I have been made to work against a moving target here. Every time I ask for feedback, I am given a list of items to fix. When I am done fixing them, a new list of "problems" comes up. A lot of the changes I made already weren't even about "right" or "wrong" but only about preferences and I want you to see that I have been cooperating so far by adjusting to your preferences rather than exert my own. But this has to end somewhere. You have to give me a final list of things to fix after which this PR would be considered ready. Instead of justifying what these tests would prove, your answer is "its a matter of 5 minutes"?
I disagree that we have achieved reproducible science in Devito btw. If we did, you would have provided me clear instructions when I asked you earlier in this thread about how I should benchmark. Or atleast you would have mentioned thread pinning in your reply. You essentially asked me to start from scratch. If every single person in the project has to reinvent the wheel to benchmark, that's hardly a sign of reproducible science.

for the future: if rebasing against my PR causes troubles, please talk to me and let's do it together. 4 hours is a lot of time for having moved just a few functions, you should have definitely asked for clarifications.
Now, about your comments, I have two things to say:

you're taking it too personal, while whatever I say is for the sake of the group/project. I might be wrong sometimes, and I'm happy to civilly discuss about everything, including my mistakes/feedback/comments/picky attitude/whatever. We can even talk over skype/slack, if you prefer.

unfortunately, you're wrong again about reproducible science, for three reasons.
2A- If you scroll up, you may read the following "Secondly, to facilitate experimentation, you can temporarily cherry-pick on your branch this commit 4b0f4c7, which gives you a launcher.sh script that you can invoke (examples folder) to run various types of experiments. If you use this, it becomes easy to reproduce exactly the same experiment." This is very clear to me. In the launcher script, everything is set, including thread pinning.
2B- Reproducible science comes in many flavours. One of them is that the numbers you read at the end of a simulation are correct, solid, and generally reproducible across a range of architectures. When you launch a test, and at the end you see something like:
Time: 1.903779 s (2516 MCells/s) Stencil: 1.687500 OI, 69.33 GFlops/s (time: 1.901353 s)
These numbers were subjected to only God knows how many bugs (they were completely broken), which I fixed, and then double-checked, profiled, etc.
2C- The benchmark.py -- that's a another symptom of reproducible science, which was possible with the help of Michael and Vincenzo over the last couple of months.
Overall, I think we are in good shape from that point of view -- perhaps far from ideal, but definitely not as bad as it seems you found

I hope this clarifies everything.

Ah, the most important thing: I'll be more than happy to give my blessing once I see the times with tiling ON.

Ok, so I think we all agree that there are a lot of very important and good things in this PR that should land ASAP, while some other things are still under discussion and require further performance testing/verification. I therefore propose a divide-and-conquer approach where we split this PR into several sub-changes, so that we

a) get a clear picture which fix has what effect
b) keep the maintenance burden low in case certain parts of this PR need to be rebased again

In particular I think the deep-copy fix in the caching and the alignment via posix_memalign are more or less universally accepted at this stage, so they each warrant their own PR and should go in unopposed. The First Touch loops itself might depend on other planned features like a global parameter dictionary, if First Touch + blocking/AT shows to be incompatible, or we might even want to internalise thread pinning before creating indisputable evidence that our first touch allocation has no size effects.
As for the benchmarking, I don't think a launcher script in a dev branch is particularly helpful. So, would it make sense to simply document the exact command for running a kind of "baseline" test in a README in the examples directory to facilitate easy before-and-after tests for individual changes? Again, internalising thread-pinning would again help make this as pain-free as possible for future changes.

Yes, we definitely need a README on how to do benchmarking. The launcher script is just a shorthand that I've used over the last few weeks -- and it's so simple that I told @mloubout and @navjotk they could have used it, temporarily. The other possibility is to extend the launcher and make it something like a "demo" that people can quickly run or take inspiration from.
Back to the PR: what I was trying to say is that one or a few tests are rarely enough to claim something about the achieved (and achievable) performance. Producing a roofline should be pretty straightforward with the current infrastructure (if not, then we have an issue to fix)... but I'd be happy even just with raw timings from one or two tests with tiling/autotuning. Even though tiling seems to sit on an orthogonal dimension. With that, for me, the PR is probably mergeable as is (although splitting isn't a bad idea, but it'd take additional time).
I also believe that this whole discussion has significantly improved and strengthen the PR, so I'd be positive about it :)

On discussing with @mloubout , it was clear that the effects of autotuning only show up on domains of size > 512x512x512. The following are runs with autotuning and cache blocking on, on both branches:
On master:
[navjot.kukreja@service8 devito]$ time KMP_AFFINITY="granularity=fine,proclist=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],explicit"  python examples/benchmark.py -P acoustic -d 512 512 512 -o -a
AutoTuner: Start. Mode: brute force
AutoTuner: Number of block sizes that will be attempted: 3
IntelCompiler: Compiling /tmp/devito-1078/1b85fc83eb77f748319a55bbf439c3c7c40be623.cpp
Stencil: 1.687500 OI, 36.80 GFlops/s (time: 0.331378 s)
Stencil: 1.687500 OI, 41.68 GFlops/s (time: 0.292582 s)
Stencil: 1.687500 OI, 41.29 GFlops/s (time: 0.295369 s)
AutoTuner: Finish.
AutoTuner: Estimated runtime for ForwardOperator and 149 time steps: 0.004037 hours
Applying Forward
AutoTuner: Picked: [6, 6, None]
IntelCompiler: Compiling /tmp/devito-1078/67b6046dd79e33b517b83741341547e51ea7c38a.cpp
Shape: (532 x 532 x 532) - Cache Blocking: [6, 6, None]
Time: 19.508382 s (1137 MCells/s)
Stencil: 1.687500 OI, 31.05 GFlops/s (time: 19.505693 s)

real	21m46.799s
user	10m18.481s
sys	25m22.133s

On this branch:
[navjot.kukreja@service8 devito]$ time KMP_AFFINITY="granularity=fine,proclist=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],explicit"  python examples/benchmark.py -P acoustic -d 512 512 512 -o -a
IntelCompiler: Compiling /tmp/devito-1078/9bef9362e33099c89a81c8433a8c3db71a3947f6.cpp
IntelCompiler: Compiling /tmp/devito-1078/a7e646699ecddf6764e74a0ab56de05d981c9d9f.cpp
IntelCompiler: Compiling /tmp/devito-1078/ad7779c9e2a14d50176ce11de301c7921a2a4fed.cpp
IntelCompiler: Compiling /tmp/devito-1078/986abc438929c93cb5712ef476c0eaae2167dc4b.cpp
IntelCompiler: Compiling /tmp/devito-1078/6c7de65a680602ddf24e2b01b1c2abe18466158a.cpp
AutoTuner: Start. Mode: brute force
AutoTuner: Number of block sizes that will be attempted: 3
IntelCompiler: Compiling /tmp/devito-1078/defc3fb56d861c5adc4ca27a6114dc6ce5f24cc3.cpp
IntelCompiler: Compiling /tmp/devito-1078/eb53275a50a1054e8a2ece64a63c37b289fbefc4.cpp
IntelCompiler: Compiling /tmp/devito-1078/aae0e4900e129e480149373d44cf431e709371f9.cpp
Stencil: 1.687500 OI, 83.02 GFlops/s (time: 0.146908 s)
Stencil: 1.687500 OI, 87.35 GFlops/s (time: 0.139626 s)
Stencil: 1.687500 OI, 83.40 GFlops/s (time: 0.146238 s)
AutoTuner: Finish.
AutoTuner: Estimated runtime for ForwardOperator and 149 time steps: 0.001927 hours
Applying Forward
AutoTuner: Picked: [6, 6, None]
IntelCompiler: Compiling /tmp/devito-1078/c106798c58ea6b4ef612e9507ed48c5a34e91402.cpp
IntelCompiler: Compiling /tmp/devito-1078/f35a00aee7a39fb37cc3401b70f46903ab065f7a.cpp
IntelCompiler: Compiling /tmp/devito-1078/011d5534b980daa610b191f31b5f0c6f206f02f9.cpp
IntelCompiler: Compiling /tmp/devito-1078/6253df18c9f885331b5579b8d6d4dfa1d7a280f9.cpp
IntelCompiler: Compiling /tmp/devito-1078/26b7520568eda7c3ec7700dea66c876b5a8a1b1f.cpp
Shape: (532 x 532 x 532) - Cache Blocking: [6, 6, None]
Time: 7.054673 s (3144 MCells/s)
Stencil: 1.687500 OI, 85.89 GFlops/s (time: 7.052508 s)

real	1m23.088s
user	9m52.758s
sys	1m37.960s

@FabioLuporini I did these tests even though your best justification for them was "it takes 5 minutes" (which wasn't even true - look at the times above. The one on master takes 22 minutes for a single run). For me these tests were no better justified than studying the effects of the latitude and longitude of the server location on performance. Do you want me to try that next? :)
I don't understand what additional information would a roofline provide, when the discussion is about the relative performance between two versions of the same code. System specifics don't matter when both versions are being run on the same machine. So again, I don't understand the relevance of what you are asking.

It takes 5 minutes to set up (less perhaps), then I guess you don't have to stare at the screen for another 22 mins
The roofline would have shown how much your fix makes us closer to the peak
One last time: you had a (reasonable) hypothesis that your optimisation is orthogonal w.r.t. to the way the loops are scheduled; I could have bought it, but I prefer to have (compelling) evidence.
I think this can be merged now.

@FabioLuporini small detail, but the default behavior of AT is not set as 'ideal_with_Stores' as @mlange05 and I asked to have 'optimal' performance tests.

@mloubout Actually upon deliberation with @FabioLuporini I had changed my mind on the default AT behaviour. The key thing I was missing at first was that ideal and ideal_with_stores only consider access to TimeData symbols, ie. u and v, but not DenseData symbols like damp. As a result these modes enforce the ideal cache assumption across the time loop, which means they assume damp and m are fully stored in cache. I don't think we want that because all our current optimisations only work on the space loops. We make the same ideal cache, but only in space assumption in the performance prediction paper.

As for the PR, I am happy for this to go in as is now. However, I am not particularly happy about how this came about. So, for the future, I believe we we should make sure that:
a) Every PR that touches performance behaviour is accompanied by compelling benchmarking evidence. We now have the infrastructure to re-run a basic set of performance tests with minimal developer effort, so there is no reason not to do this. We probably do want to note a sample configuration in a README though.
b) We break up large features into separate PRs for individual components. Branching is cheap in git and pushing many small changes will make review and discussion much easier for everybody. If a feature becomes large enough to warrant a long list of PRs we can use github issues to track the overall development of the feature. The pain of not doing so is clearly evident in this PR: We have at least three individual performance fixes/features (deep-copy fix, C-level alignment and first touch loop), yet we have absolutely no indication of which fix/feature has what level of impact on the overall performance improvements presented by the author. We have also held an important and undisputed set of fixes for a long time for no other reason than the author insisting on these fixes/features to be treated in bulk with several contested changes. Stating that the bulk improvement of a complex set of changes is positive overall is not sufficient to ignore potential problems with individual components!
As a final illustration why this type of bloated bulk PR is bad engineering, it also introduces a completely new feature (n-dimensional space loops) without any explicit testing of the new feature! So, while this is ok for now, I would hope that this is the last one that forces us to overlook these problems for the sake of overall progress.

grumble, grumble ... merging ... grumble, grumble, ....


Update Fauxhai mocks in Chefspec

Use non-deprecated mocks.
Signed-off-by: Tim Smith<EMAIL_ADDRESS>

No DCO necessary on those non-IP changes


Change Email on the contacts page - For OSDW 2017

Change the email to<EMAIL_ADDRESS>

This issue was moved to kashifzaidi1/Messiah-Web#2


STEP I/O

As long as we claim this crates a CAD Kernel, STEP I/O is unavoidable.
We're working on a plan for this, which we'll announce here soon.

I'm currently working on an EXPRESS to Rust transpiler and STEP parser with the aim of extracting structural and topological information. I was planning on also implementing a basic CAD Kernel to achieve this, until I stumbled across your amazing project. Perhaps we can collaborate on this.

@g-rauhoeft
Thank you for your kindly suggestion. Now, we are also working on an EXPRESS to Rust transpiler and STEP parser. That is still WIP, but we have decided to make the code public in response to your suggestion.
https://github.com/ricosjp/ruststep
This one is purely a transpiler and parser, so it is maintained in a separate repository. Most of the expansion of truck for STEP will be done pending completion of ruststep. We are waiting for your advice, issues or pull requests for both truck and ruststep!

Of course, since truck and ruststep are independent of each other, truck can support several STEP utilities with different backends.

I also did some work on reading STEP files. https://github.com/J-F-Liu/iso-10303

ruststep has been released!

ruststep has been released!

Congratulations and thank you! It's already been a pleasure to work with it and I'm looking forward to contribute more in future. :)

I'm currently working on STEP importing. Is it ok for you if I make a pull request once I'm done or am I interfering with your work/timeline?


[RFC] Use clang-format code style

What do you guys think about having a common code style using clang-format?
This is a first draft, if you spot some problematic pattern we can check whether we can tweak the settings.
@filipnavara @qmfrederik @hughbe

In general I am fine with whatever style you prefer as long as VS Code can enforce it for me. It seems that there is an extension for that (https://marketplace.visualstudio.com/items?itemName=xaver.clang-format) but I didn't try it yet.

Same here. I'm all about enforcing a common style, less opinionated about which style that should be :)

Happy with this


A general discussion on coding conventions

This is to start the ~troll~ debate on coding conventions in Radium.
I encourage you to read the current convention (https://github.com/AGGA-IRIT/Radium-Engine/blob/master/conventions.md), and to discuss and argument if anything should be added, deleted or changed.
Here are the main rationales behind the decisions we had earlier in the project.
Angle brackets in includes
First, I think include paths should never be defined relatively. This is obviously wrong:
#include "../../foo.hpp"

I argue that even this example is bad
#include "foo.hpp"

There is a semantic value in the full path of an include from the project's root (this supposes that the directory hierarchy makes sense, of course). There is more info in such a declaration than in the previous one :
#include "Core/Math/Foomatic/foo.hpp"

With files having dozens of includes this helps ordering them and identifying which ones might be useless.
So, now that I established that, from my point of view it is preferable to use the full paths, why force <> ?

Because it enforces the rule of no local paths (#include <foo.hpp> will not work)
Because it makes a simple and easy to remember rule.

Brace style
Why the Allman style you say ?

We now have big screens so we can spare a line or two
In my experiences it allows easier modification of code e.g. when commenting you control structures

// if ( what() )
{
    doIt();
    doItNow();
}

Is still syntaxtically correct.
This also makes sense in the context of enforcing braces everywhere. The one-line if has been the source of many bugs. The canonical example is when you have
/// this is bad.
if (blah)
   doX();

and you add  another function
if (blah)
   doX();
   doY(); // oh, no !

Of course doY() is always executed regardless of the if. This is why I wish to enforce braces
even for one-line if.
/// this is better
if (blah) {
   doX();
}

I argue that it's easier to visually scan for such constructs if you forces matching braces below every if statement.
/// this is even better
if (blah)
{
   doX();
}

Tabs vs spaces
Endless years of religious wars have not settled this debate.
In an ideal world we would use tabs for indentations and space for alignments. The world is not ideal. Thus, it's just simpler to just use spaces everywhere.
No exceptions, RTTI
Writing exception-safe code requires effort and is still awkward to use with low-level codes (which uses return code based errors), and we don't need no fancy error handling.
RTTI is a bit more controversial. I argue that RTTI (especially dynamic cast) encourages bad design  and it adds a small penalty (more memory for each class) so it's not worth using in general.

My name is Hebus .. Troll Hebus ... So, yep, I'll troll.
I agree that
#include "../../foo.hpp"
is a nightmare and must be prevented.
but I think that
#include "foo.hpp"
is very clear and indicates, "include the header foo from the current package"
while
#include "Core/Math/Foomatic/foo.hpp"
might be interpreted as "include the header foo from the package Core/Math/Foomatic/ one may found in the implementation dependant location of search.
In the C standard, section 6.10.2, paragraphs 2 to 4 state:

A preprocessing directive of the form
#include <h-char-sequence> new-line
searches a sequence of implementation-defined places for a header identified uniquely by the specified sequence between the < and > delimiters, and causes the replacement of that directive by the entire contents of the header. How the places are specified or the header identified is implementation-defined.

A preprocessing directive of the form
#include "q-char-sequence" new-line
causes the replacement of that directive by the entire contents of the source file identified by the specified sequence between the " delimiters. The named source file is searched for in an implementation-defined manner. If this search is not supported, or if the search fails, the directive is reprocessed as if it read
#include <h-char-sequence> new-line
with the identical contained sequence (including > characters, if any) from the original directive.

A preprocessing directive of the form
#include pp-tokens new-line
(that does not match one of the two previous forms) is permitted. The preprocessing tokens after include in the directive are processed just as in normal text. (Each identifier currently defined as a macro name is replaced by its replacement list of preprocessing tokens.) The directive resulting after all replacements shall match one of the two previous forms. The method by which a sequence of preprocessing tokens between a < and a > preprocessing token pair or a pair of " characters is combined into a single header name preprocessing token is implementation-defined.

Definitions:
h-char: any member of the source character set except the new-line character and >
q-char: any member of the source character set except the new-line character and "
So, one can choose <> or "" to include files.
I found a quick advantage of using "" rather than <> for including files related to the current project.
This allow to have a quick identification of standard dependance (included with <>) and intern dependance (included by "").
Note that "" does not prevent to use a clean identification by Module/Submodule/CaptainIgloo/...
As we are on the header topic and inclusion, I agree with dots 1 and 2 (even if the name .hpp seems ugly) but I'm dubious about the 3rd. Why not use #pragma once instead of a guard that assume one know all the guards defined in a project ? #pragma once is supported by a lot of compilers (see https://en.wikipedia.org/wiki/Pragma_once ) and much less error prone.
I agree with the point 4, but mention that it is not so well respected in the current code - this cause problems for me while integrating glbinding.
For point 5 refer to the discussion above.
For Point 6, I strongly disagree as soon as point 7 is not mandatory as well as the missing point 8 saying "Only include what is needed and not the whole universe" I reach a ton of problems simplifying the include graphs wile integrating glbinding to separate Qt and its somewhat crappy OpenGL management from core OpenGL. Integrating glew with Qt, for those who experienced this, is wayyyyyy more easy than glbinding. But glbinding is wayyyyyy more efficient and usefull than glew.
Keep listening on this channel, soon the Troll Akläss will discuss about Functions, Scope, names, variable, code style and code design.
Cheers (or Roaaaaaarrrrrrrhhhhhh depending of your Troll Itude)

:trollface:

#include "foo.hpp"
is very clear and indicates, "include the header foo from the current package"
while
#include "Core/Math/Foomatic/foo.hpp"
might be interpreted as "include the header foo from the package Core/Math/Foomatic/ one may found in the implementation dependant location of search.

I agree that the policy to use quotes for local code makes sense given the language definition,
however, this raises the question of what exactly is a "current package". Is it the whole project ? The library ? (Core / Engine / Guibase / etc) ? The module inside the library ? (e.g. Engine/Renderer) ? The current folder ?
Let's say we are in the implementation of class Toto in library C.
Currently this looks like this:
#include <C/Toto/Toto.hpp>

#include <cstring>
#include <A/Foo/Foomatic.hpp>
#include <A/Bar/Bar.hpp>
#include <B/Twiddling/Twiddler.hpp>
#include <C/Frob/Frob.hpp>

If I assume "current package" means the current library, here's what it  will look like :
#include "Toto.hpp"

#include <cstring>
#include <A/Foo/Foomatic.hpp>
#include <A/Bar/Bar.hpp>
#include <B/Twiddling/Twiddler.hpp>
#include "Frob/Frob.hpp"

Is that what you have in mind ?
I could agree to this (but the inconsistency between the last line and the other is kind of annoying).

As we are on the header topic and inclusion, I agree with dots 1 and 2 (even if the name .hpp seems ugly)

C files : .c, C header files .h
C++ files .cpp, C++ header files .hpp
what's not to like ?

but I'm dubious about the 3rd. Why not use #pragma once instead of a guard that assume one know all the guards defined in a project ? #pragma once is supported by a lot of compilers (see https://en.wikipedia.org/wiki/Pragma_once ) and much less error prone.

I agree, we could use #pragma once. It seems the benefits outweights the potential risks.
Further discussion on the topic can be found on the interwebs :

http://wiki.c2.com/?RedundantIncludeGuards
https://www.reddit.com/r/cpp/comments/4cjjwe/come_on_guys_put_pragma_once_in_the_standard/d1j04te/

For Point 6, I strongly disagree as soon as point 7 is not mandatory as well as the missing point 8 saying "Only include what is needed and not the whole universe" I reach a ton of problems simplifying the include graphs wile integrating glbinding to separate Qt and its somewhat crappy OpenGL management from core OpenGL. Integrating glew with Qt, for those who experienced this, is wayyyyyy more easy than glbinding. But glbinding is wayyyyyy more efficient and usefull than glew

I wholeheartedly agree. We need to enforce forward declaration as much as possible.
On the other hand, let's say your class Toto need a instance of class Foo and class Bar.
It just so happens that currently, Bar also needs a Foo instance, so bar.hpp includes foo.hpp. Does it make sense to include just bar.hpp or to include both ?
The argument for including both is that it is only incidental that Bar depends of Foo and in the future, a revision of Bar could remove this dependency. Having one include would make us modify the header of Toto which shouldn't be related.

So, is there a proposition to change the coding convention or should I just close this issue for now ?

BTW, we should define a common coding style, and I propose to use an automatic code formater (to have the same indentation, tab, openbrace style).
Either clang-format or astyle is fine, and with a predefined style is simplest
My opinion is to have a compact style mode ;) but I don't really care
Check
http://astyle.sourceforge.net/astyle.html#_Brace_Style_Options
for instance Google style.

Independently of the style we use, I suggest we call the code formatter automatically as commit pre-hook (https://gist.github.com/wangkuiyi/7379a242f0d4089eaa75).
Of course, any developer should also configure properly is IDE to use the style definition.

There is a astylercfile in the root of the Radium folder hierarchy (although I doubt it has ever been used).
I would be all in favor of a pre-commit formatting (but we have to make sure that it works on Windows too !)

Cording guidelines need to be clearly explained (and the process to use it with hooks and IDE) as suggested here: https://help.github.com/articles/setting-guidelines-for-repository-contributors/

FYI: astyle can be used with Visual Studio, but not directly using the file today provided.

I'm using Clion as IDE. Clion proposes strong formating facilities (pre-commit, on the fly), using various predefined style. I've not found, yet, how to integrate astyle definiton but I can quite easily parameterize the IDE to follow what I understand in the astylerc. I've just one question : why CRLF at line end ? ...

I propose to use clang-format with standard style (say llvm), it will ease a lot settings of each ide, and do not change our habits ;)
I could commit a set of script one can add as pre-commit hook to format their files (at least for linux/macos)

+1 for llvm style.
+1 for your hooks ;)

+1 LLVM.
The style is clearly explained and justified at this URL.
http://llvm.org/docs/CodingStandards.html


[ENH] Retry url request after HTTPError

Reference Issues/PRs

Follow up on Slack conversation: "Looks like test_load_solar is causing CICD to fail. Does anyone experience something similar?" (2022-08-11 in Contributors channel). More precisely, here is the HTTPError causing the failure: https://github.com/alan-turing-institute/sktime/runs/7774941244?check_suite_focus=true#step:10:1676
What does this implement/fix? Explain your changes.

Use backoff library to retry data load from url in sktime.datasets._single_problem_loaders.load_solar.
Does your contribution introduce a new dependency? If yes, which one?

backoff
What should a reviewer concentrate their feedback on?

Does this failure happens so often that it is worth introducing a new dependency?
Any other comments?

PR checklist

For all contributions

[ ] I've added myself to the list of contributors.
[ ] Optionally, I've updated sktime's CODEOWNERS to receive notifications about future changes to these files.
[X] I've added unit tests and made sure they pass locally.
[X] The PR title starts with either [ENH], [MNT], [DOC], or [BUG] indicating whether the PR topic is related to enhancement, maintenance, documentation, or bug.

assuming you are on holiday or otherwise off, @khrapovs, so I´m just moving this to the other dependency set myself (to avoid the load errors)

I wanted to come back to it on Monday, but you are welcome to take it over if you think it becomes a pressing issue.

I wanted to come back to it on Monday, but you are welcome to take it over if you think it becomes a pressing issue.

ok, sorry - feel free to revert my changes.
I was trying to isolate backoff as a soft dependency, as moving it to the soft dependency set was breaking things here and there.

merging since the error is popping up frequently on PR


Replace dash with em dash to lessen confusion of negative scores

Replace dash with em dash to lessen confusion of negative scores

Awesome change!! I honestly didn't have a solution for this one.
Also, you're other PRs are very helpful. Thanks for spending time on this.


Fields of Issue are private

Hi,
I'm able to get and filter Issues with their library but how can I do something with the object when all its fields are private?
Thank you

It shouldn't be; if they are private, it's a bug ;)
Fields are public on master but they are not on v0.4.0 which is the version crates.io.
I wanted to release a v0.5.0 with some breaking changes (this is one of those) but did not have time to finalize the release.
The next two weeks will still be quite busy, I'm not sure I'l have much time working on this until then. Meanwhile you can always point master the master branch:
gitlab-api = { git = "https://github.com/nbigaouette/gitlab-api-rs" }

Oh, I didn't realize that it's different on master. Also, thanks for showing how to directly use the repo with cargo.

Just wanted to let you know that version 0.5 out!
Should now be as simple as:
gitlab-api = "0.5.2"

in your Cargo.toml.
Enjoy! ;)

Awesome! I'll try it out immediately. Thanks :)


Using Password in functions is not secure. Implement zk commitments or alternative way.

Like tornado cash, implementing zk commitments with circom can be used provide security.

Love that. I'm mostly busy with my job but will look into it soon.