config: full delta xDS (including ADS) support

Delta ADS/xDS support. Added NewGrpcMuxImpl, a GrpcMux implementation that supports delta (and is intended to soon take over for state-of-the-world). DeltaSubscriptionImpl (which can soon be renamed to/replace GrpcSubscriptionImpl) now uses that new class. NewGrpcMuxImpl is the first user of the recently added WatchMap.
Risk Level: medium (huge new thing that people aren't using yet)
Testing: added delta param variant to ads_integration_test
Docs Changes: added source/common/config/README.md and accompanying diagram
Release Notes: added support for delta xDS (including ADS) delivery
Fixes #4991

This PR is not quite as big as it looks. First, most of the 86 changed files are only changed to have the bool is_delta plumbing, which is just a temporary measure until NewGrpcMuxImpl unifies implementation of delta+SotW. (Which will be a fairly simple and immediate followup to this PR). Second, WatchMap is not yet merged (#7108).

Quick drive by: would it be possible to add something about delta vs. SoTW to the arch overview? Maybe here? https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration? Maybe also link to the detailed xDS API specification for more details? I think it would be useful to spruce up this page a bit just to talk at a high level about the various options. Thank you!

Yes, great idea, will do!

Will review once we have #7108 design level agreement.
/wait

/nostalebot

At some point can you do a master merge, and move your release note to 1.12.0?  Thanks!

Yup, done.

Reiterating so it doesn't get lost in the shuffle: many of the vast number of files changed only had the "is_delta" param added. That param is an ugly hack to allow this PR to leave the unification of delta+SotW for later. That unification will remove the is_delta from everywhere, and is ready to happen immediately after this PR. (Or even within this PR if you want to do a sort of multi-stage review).

Thanks! Also, reiterating again so it doesn't get lost in the shuffle:
Many of the vast number of files changed only had the "is_delta" param added. That param is an ugly hack to allow this PR to leave the unification of delta+SotW for later. That unification will remove the is_delta from everywhere, and is ready to happen immediately after this PR. (Or even within this PR if you want to do a sort of multi-stage review).

@lizan
As I've mentioned a couple of times, this PR is currently a bit messy - the is_delta argument all over the place - due to keeping the work needed to unify delta and state-of-the-world, out of this set of changes.
However, that unified state is the end goal, and I think it has a more natural flow. In fact, I'm starting to question whether it was even worth trying to present the "simpler" delta-only version here.
To speed this process up, and avoid having a whole second PR to be done when this one finally merges, I have merged my long-standing rough drafts of the unified stuff (GrpcMuxImpl, SubscriptionState) into the actual code. There were some conflicts that built up over the past few months, but I've got them mostly worked out; it's passing the main part of ads_integration_test. It is over at https://github.com/envoyproxy/envoy/compare/master...fredlas:unified_ADS
You may still find this current PR easier to review; not sure. But, if you do find the new one easier, let me know, and I'll just merge it straight into here.

Oh! Almost forgot: in the unified version, there are some changes made to naming.

The existing GrpcMuxImpl, GrpcSubscriptionImpl, GrpcMuxSubscriptionImpl have been deleted.
The DeltaSubscriptionImpl seen in this PR has been renamed to GrpcSubscriptionImpl; it now also supports state-of-the-world (taking over from GrpcSubscriptionImpl and GrpcMuxSubscriptionImpl).
The NewGrpcMuxImpl seen in this PR has been renamed to GrpcMuxImpl.
DeltaSubscriptionState has had a semi-abstract "SubscriptionState" base class factored out of it, so that there can also be a SotwSubscriptionState that GrpcMux can use interchangeably.

@fredlas I'll review this again later today, can you merge master to make CI green?

Thanks, done.

Well, the tests all passed, but clang_tidy had a couple of issues; should be fixed now. Actually, it had many issues, all in server.cc, but IIUC the issues that are pre-existing in parts of the code I'm not modifying won't be blockers.

LGTM!

Thanks very much! It is great to have it in. I will try to get the delta+SotW unification in swiftly, so we will have nice clean, modular gRPC xDS code, more amenable to future changes and additions.

@fredlas Does the api doc need update?
DELTA_GRPC is not yet entirely implemented! Initially, only CDS is available. Do not use for other xDSes. TODO(fredlas) update/remove this warning when appropriate.

https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/core/config_source.proto#enum-core-apiconfigsource-apitype

@fredlas Does the api doc need update?
DELTA_GRPC is not yet entirely implemented! Initially, only CDS is available. Do not use for other xDSes. TODO(fredlas) update/remove this warning when appropriate.

https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/core/config_source.proto#enum-core-apiconfigsource-apitype

Looks like it, thanks for pointing it out! I'll add that to #8478

I have read the relevant documentation and code, but I still don't know how the lazy loading and other functions are implemented. I didn't find the code to update the 'watch' at runtime. Or is this just an infrastructure, and specific features such as lazy loading are not implemented?

Correct, lazy loading is for later. That was actually the motivation behind renaming all of this to "delta", to emphasize the aspect it actually includes, while leaving "incremental" to describe the eventual inclusion of lazy loading. (Which will also need to reach way into Envoy, through the various xDS users, all the way to the data plane, and generally be an even bigger change than this one, I think).


Build Odoo 18 images

Hm, let's stick to jammy as I don't have time to investigate if wkhtmltopdf is available for noble.

And postgres 9.6 is too old for Odoo 18 :) So let's use the oldest officially supported, which is postgres 12.


chipCheckedChange Listener called for every chip every time

Whenver I click on any chip, chipCheckedChangeListener is first called for that chip with checked = true.
chipCheckedChangeListener is then called for every other chip with checked = false.
Is this intended behaviour?

For anyone else who runs into this problem: After digging around the source code it appears there is an overload of setListener that takes the parameter ignoreAutoChecks. If you set this to true you will only get a callback for the selected chip.
I'm curious however as to why this isn't the default.


Section DP-5: Encrypt sensitive data at rest

Section DP-5: Encrypt sensitive data at rest is out of date. Please update. Purview is now GA.
Currently, Azure Purview doesn't support encryption at rest of its metadata with a customer-managed key. This feature is planned for when the service becomes generally available (GA).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 195719c8-bc66-4235-0cd3-73cd989c7c45
Version Independent ID: 16847b71-4999-a446-75ff-abe8804a5624
Content: Azure security baseline for Azure Purview
Content Source: benchmark/azure/baselines/purview-security-baseline.md
Service: purview
GitHub Login: @msmbaldwin
Microsoft Alias: mbaldwin

@isdataninja, Thanks for bringing this to our attention! We are taking a look into this and will get back to you soon.

@isdataninja - Thank you! We are updating this now.


Remove path filters for fast tests

What does this PR do?

This PR is necessary to enable "Require status checks to pass before merging" as filtered workflow will be waited for forever otherwise. More information and potential solutions here: https://github.com/orgs/community/discussions/13690
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


[Correction] page 39 "General Journal" - OnAction PostAndPrint

action(PostAndPrint)
{
trigger OnAction()
begin
    SendToPosting(Codeunit::"Gen. Jnl.-Post+Print");
    CurrentJnlBatchName := GetRangeMax("Journal Batch Name");
    if GeneralLedgerSetup."Post & Print with Job Queue" then
        NewDocumentNo()
    else
        SetDataForSimpleModeOnPost;
    SetJobQueueVisibility();
    CurrPage.Update(false);
end;
}

Where is the line "IF IsSimplePage THEN" present in the old NAV versions ?

BC140 :

PostAndPrint - OnAction()
CODEUNIT.RUN(CODEUNIT::"Gen. Jnl.-Post+Print",Rec);
CurrentJnlBatchName := GETRANGEMAX("Journal Batch Name");
IF IsSimplePage THEN
  SetDataForSimpleModeOnPost;
CurrPage.UPDATE(FALSE);

We will add an onbefore event to SetDataForSimpleModeOnPost with IsSimplePage as param so you can decide for yourself if you want to settimple mode or not.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Support legacy accounts

What is the expected enhancement?
Support legacy accounts (no cloud accounts). Copy account management related code from qiskit-ibm-runtime.

I'll mark this done (fixed via https://github.com/Qiskit/qiskit-ibm-provider/pull/259). Follow-up efforts are covered via

https://github.com/Qiskit/qiskit-ibm-provider/issues/256
https://github.com/Qiskit/qiskit-ibm-provider/issues/262


64-bit Libraries iOS

Expected behavior
Actual behavior
Steps to reproduce the behavior
Please include enough details so that the issue can reproduced independently by the resolver.

Hi,
I am trying to compile for iOS 12 and need 64-bit libraries for libpv_porcupine.a  Do you have those

changed the title to 64-bit ... I don't think anyone will build a 65-bit ... CPU anytime soon.
The lib works on iOS 12. Be sure to build for target and not the simulator, please.


Use of IList

Several types use IList<T> to expose collections of data. ClrType is a good example.
There are a few issues with this:

Enumerating them requires allocating an enumerator
They're mutable
Calls are virtual

The mutability thing is not a problem when everyone is well behaved, but the allocation/virtual points are only avoidable by casting to List<T> or T[], which is suboptimal and risky.
Would you consider a dependency upon System.Collections.Immutable and using ImmutableArray<T> in place of IList<T>? When used well, it provides the performance and compactness of arrays with immutability and essentially zero overhead. It would be a breaking API change however.

When was ImmutableArray added?  Currently ClrMD is supported as far back as CLR 4.5 (and needs to be carried without dependencies on Desktop CLR).  Was ImmutableArray around in 4.5?
I do wish I hadn't used IList so long ago, but I worry it's too big of a breaking change to modify these return types.

I'm pretty sure it's always been distributed as a NuGet package. The latest release supports net45.
https://www.nuget.org/packages/System.Collections.Immutable
https://github.com/dotnet/corefx/blob/master/src/System.Collections.Immutable/src/System/Collections/Immutable/ImmutableArray_1.cs
ImmutableArray<T> does implement IList<T> which would ease a transition, however it's a struct which means boxing and, more significantly, no null value (you use IsDefault or IsDefaultOrEmpty properties). For APIs that can never return null this might not be such a big deal.
I get the importance of backwards compatability.
If it's something you wanted to try out, I would be willing to help.

That runs into the second part of the problem, ClrMD on desktop CLR is supposed to run without any dependencies (not having to carry other DLLs).  There's still a small window of time we could probably make breaking changes for 1.0 of clrmd (especially since I need to do a large round of bugfixing).  Let me think about this.

If it helps, the actual implementation of ImmutableArray is very simple and could be easily replicated within ClrMD directly to avoid any dependencies. There's no need for the other immutable collection types if you just want the immutable array container. Again, I'm happy to spike this out if you feel it might be worth it. No worries either way.

Ah good point.  Sure, if it's not too much work.  Let's see what this looks like putting an immutable array type into ClrMD.  I wouldn't mind the change so much if it implements IList to ease the transition.

Fixed in ClrMD 2.0.

Fantastic, thanks.

It seems that the fix was to change IList<T>s to IReadOnlyList<T>s, so the issues with enumerator allocations and virtual dispatching still remain. Since ClrMD 2.0 can carry dependencies, how do you feel about ImmutableArray<T>?

I'd be happy using ImmutableArray.  I'll reopen this issue to track it.


Migration to V3 -> issue with regenerating OData Client

Describe the bug
After upgrading to V3 of the SAP Cloud SDK, I try to regenerate the OData Client via CLI. To the best of my knowledge all adjustments in the project are made.
Executing the generation with the setup working in V2 fails with V3 as an error is raised that the configuration is invalid.
To Reproduce
Steps to reproduce the behavior:

Clone the repository https://github.com/lechnerc77/app-service-javascript-sap-cloud-sdk-quickstart
Switch to branch issue-46
Navigate to the directory src/api
Execute a npm i
Execute the generation via:

Option1: npx generate-odata-client --input service-specifications/ZAPI_BUSINESS_PARTNER.edmx --outputDir services -c service-specifications/config.json --clearOutputDir => this will lead to the error Unknown argument: API_BUSINESS_PARTNER which is part of the configuration
Option 2: npx generate-odata-client --input service-specifications/ZAPI_BUSINESS_PARTNER.edmx --outputDir services -c service-specifications --clearOutputDir=> this should pick up the config.json in the specified directory. It raises the errorInvalid JSON config file: service-specifications`

The configuration file is consistent with the one described here
Expected behavior
Generation of the Client should be executed successfully.
Screenshots
Used Versions:

node version via `node -v': v18.14.2
npm version via npm -v: 9.5.1
SAP Cloud SDK version you used as dependency: 3.0.1

Code Examples
See above (https://github.com/lechnerc77/app-service-javascript-sap-cloud-sdk-quickstart/tree/issue-46/src/api)
Log file
--verbose does not add anymore information than the error
Impact / Priority
Demo for DSAG Technologietage on 22nd of March
Additional context
n/a

Hi @lechnerc77 ,
Thank you very much for your adoption of version 3. The example you pointed to here is a bit outdated. Thanks for pointing it out. I will correct this.
Please refer to our Upgrade Guide which shows the new format for the options for service file.
To fix the issue, update the service-specifications/config.json file manually to the new format (remove the old service options):
  "service-specifications/ZAPI_BUSINESS_PARTNER.edmx": {
    "packageName": "business-partner-service-1",
    "directoryName": "business-partner-service-1",
    "basePath": "/sap/opu/odata/sap/API_BUSINESS_PARTNER"
  }

and re-run client generation using the command below:
npx generate-odata-client --input service-specifications/ZAPI_BUSINESS_PARTNER.edmx --outputDir services --optionsPerService service-specifications/config.json --clearOutputDir
Note: the correct option for specifying the service options is --optionsPerService. -c option is for providing path to a config file that contains generator options.

You are right - looked at the right time in the wrong spot of the documentation. Thanks for the quick response and support!


ValueError: There is no such driver by url http://chromedriver.storage.googleapis.com/LATEST_RELEASE_115.0.5790

hello i am getting this error related to browser [2023-08-02 16:57:15,952] ERROR in app: Exception on /scrape-facebook [POST]
Traceback (most recent call last):
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\flask\app.py", line 2190, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\flask\app.py", line 1486, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\flask\app.py", line 1484, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\flask\app.py", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "C:\Users\HP Probook\PycharmProjects\scraping-scripts\facebook.py", line 52, in scrape_facebook_page
json_data = namepage.scrap_to_json(plateforme_name="facebook")
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\facebook_page_scraper\scraper.py", line 82, in scrap_to_json
self.__start_driver()
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\facebook_page_scraper\scraper.py", line 61, in __start_driver
self.__driver = Initializer(
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\facebook_page_scraper\driver_initialization.py", line 90, in init
driver = self.set_driver_for_browser(self.browser_name)
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\facebook_page_scraper\driver_initialization.py", line 66, in set_driver_for_browser
return webdriver.Chrome(executable_path=ChromeDriverManager().install(),
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\webdriver_manager\chrome.py", line 32, in install
driver_path = self._get_driver_path(self.driver)
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\webdriver_manager\manager.py", line 22, in _get_driver_path
driver_version = driver.get_version()
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\webdriver_manager\driver.py", line 40, in get_version
return self.get_latest_release_version()
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\webdriver_manager\driver.py", line 64, in get_latest_release_version
validate_response(resp)
File "C:\Users\HP Probook\PycharmProjects\scrap-scripts\venv\lib\site-packages\webdriver_manager\utils.py", line 79, in validate_response
raise ValueError("There is no such driver by url {}".format(resp.url))
ValueError: There is no such driver by url http://chromedriver.storage.googleapis.com/LATEST_RELEASE_115.0.5790

I'm having the same issue. Following.

Solution appears to be to update webdriver-manager:
python3 -m pip install webdriver-manager --upgrade
(from https://stackoverflow.com/a/76922601/5273071)

Hello, in the setup.py you'll fing requirements = ['selenium==4.1.0',
'webdriver-manager==3.2.2',
'selenium-wire==5.1.0',
'python-dateutil==2.8.2']
change the 3.2.2 to 4.0.1


On dual monitors, when a module has a long loading time, it will only load on one monitor.

Describe the issue
I found this with the updates-arch-combined module from https://github.com/x70b1/polybar-scripts/. When using two (or more?) monitors and using the same bar on each monitor, if there's a module that has a long loading time, polybar will only load that module on 1 monitor. In my case, with updates-arch-combined, it has a 2 second delay before appearing on 1 monitor, but after the set interval (mine is 600 seconds), the module will be loaded on my second monitor.
Expected behavior:
The module will load at the same time, on both bars.
Actual behavior:
The module loads on 1 bar, and after the interval set in the polybar config, it will load it on the other bar.
To Reproduce

Use a module that has a long loading time (like updates-arch-combined, where it needs to contact update servers).
Have 2 monitors and load the same bar on each of them.
Reload polybar. The module will load on only 1 bar.
After x seconds (where interval=x in config), the module will also load on the other bar. Set x to 10 or something to debug.

Polybar Log
Nothing in log.
Environment:

WM: bspwm
Output of polybar -vvv:

polybar 3.2.1

Features: +alsa +curl -i3 -mpd +network +pulseaudio +xkeyboard

X extensions: +randr (+monitors) -render -damage -sync -composite +xkb +xrm +xcursor

Build type: RelWithDebInfo
Compiler: /usr/bin/c++
Compiler flags: -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -Wall -Wextra -Werror -Wno-noexcept-type -O2 -pedantic -pedantic-errors
Linker flags: **-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now**

More background: https://github.com/x70b1/polybar-scripts/issues/108

More Background: https://github.com/x70b1/polybar-scripts/issues/108

A consequence of this is that information is not the same on 2 monitors.
For instance, with updates-arch-combined, one monitor might show a different amount of updates than the other monitor(s), because the information is not called at the same time.

Are you sure this happens with any long running command? I suspect that whichever bar runs first locks the pacman database and in the other bar the checkupdates command fails, so the updates-arch-combined script doesn't output anything. To confirm this, try running updates-arch-combined in two terminals side-by-side at the same time and see if one of the commands doesn't output anything

I can confirm that one checkupdates fails when running it twice at the same time. How can I make sure the second module loads right after the first one is done?

I have already talked to x70b1 about changes to the script, until then, you could change the exec key to:
exec = CHECKUPDATES_DB=$(mktemp -d) ~/polybar-scripts/updates-arch-combined.sh

This would prevent the two calls to checkupdates from interfering with each other.
Closing this as it's not a polybar issue.

I add this solution as workaround to our polybar-scripts repo.


Update tag_carbon-management.yaml

Add variables for Carbon Capture with levels of disaggregation based on the fate of the captured CO2 (i.e., geologic storage, utilization, other storage)
@strefler @fschreyer @gidden @Renato-Rodrigues @ouyang363, others please comment as needed

Looks good to me, overall. Not sure whether we already raised this somewhere but would it be worth to differentiate carbon capture into sources and sinks? Like
Carbon Capture|Source|Industry|Biomass
...
Carbon Capture|Sinks|Geological Storage
...

Do we have variables for the different sources of carbon capture already?

Just an FYI that I think we should discuss this an #80  in our upcoming call. I guess it should probably be 1 PR for both?

Closing in favor of #83


Upgrade worker to support Python3.7

Now that it's out of RC.
TBD - Investigate implications for the Azure SDK and other libraries that are only compatible with 3.6 so far.

fwiw, we're doing builds/tests of azure client lib usage with 3.7, seems to work well. https://travis-ci.org/capitalone/cloud-custodian/jobs/409066825

Update CI to test for Python 3.7
Updated underlying Docker image

Thanks @priyaananthasankar .
Posting the highlights here from standup:

Changes that @elprans made yesterday make the language worker agnostic of whether the user is on 3.6 or 3.7. These changes will go out with the next release.
Additional work item required is to update the docker image to use 3.7 instead.

We'll hold back on #2 until python3.7 is more stable.


Plentey Security Vulnerabilities in OpenSSL 1.0.1l

https://www.cvedetails.com/vulnerability-list.php?vendor_id=217&product_id=383&version_id=180646&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&cweid=0&order=1&trc=36&sha=7653fbcb234aa07e1b868fbd70946317d85ca17f
The DeepOnion Wallet use OpenSSL 1.0.1l  from the 15 Jan 2015
it is high time to update the client once.

This is something that is in progress.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/deeponion/deeponion/issues/176#event-2209550314, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AkyoZ7f88PWGxaJkpkqU9KCwH6TtSW3yks5vX0rmgaJpZM4b4jJh
.


Show how to run nRF Connect Unit tests

As this ticket shows this is not yet supported in this image. You should be able to build a custom image and install ruby there. However you might also be fine with the Zephyr Docker images to run Twister: https://github.com/zephyrproject-rtos/docker-image

Thanks for the quick response.
I'll look into the custom image option.


[cli] fast refresh

Allow reloading the changes without restarting the bot when possible

https://www.npmjs.com/package/precinct

https://www.npmjs.com/package/chokidar

We need to preserve client's connection while reloading the project. Simple fs-watch won't be enough for this.


How to put implicitlywait in onComplete method

Hi,
I am writing a custom reporter and i have to wait for IO write.  My reporter function gets trigger in onComplete method. How to implicitly put wait on onComplete?
I have tried below ways.
1.browser.manage().timeouts().implicitlyWait(20000);
2.onComplete: function() {
browser.wait(function() {
....
finalreport.generateHtmlReport();
},20000);
},
None of the above worked. Any other working solutions...
Thanks in Advance,
Raghul

There isn't any way to do that right now unfortunately, as we're limited by the test runners on what we can do (i.e. jasmine). Can you not use writesync? (http://nodejs.org/api/fs.html#fs_fs_writesync_fd_buffer_offset_length_position)

Thanks for the information. I have used writesync and it worked.


SSL flag for pg_restore

Please update the documentation to include ssl flag for pg_restore command. Thank you!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 43b2566e-1fc7-cf49-512a-794f56760672
Version Independent ID: fe824068-38f3-d7db-260f-e2af6e5fa7a1
Content: How To Dump and Restore in Azure Database for PostgreSQL | Microsoft Docs
Content Source: articles/postgresql/howto-migrate-using-dump-and-restore.md
Service: postgresql
GitHub Login: @rachel-msft
Microsoft Alias: raagyema

@skepticatgit
Thanks for the feedback! We are currently investigating and will update you shortly.

The two pg_restore examples need to show SSL as being enabled:
https://docs.microsoft.com/en-us/azure/postgresql/howto-migrate-using-dump-and-restore#restore-the-data-into-the-target-azure-database-for-postrgesql-using-pgrestore

@skepticatgit could you provide more information on your request? This is the PostgreSQL official documentation on pg_restore. Which parameter are you referring to?

#in-progress

@JasonWHowell It makes sense to enable SSL when loading/extracting data to/from a hosted data store. The official PostgreSQL documentation does not show SSL as being an option, unless I am missing something. Thank you.

@skepticatgit The SSL functionality is implemented via an environment variable and not passed as a command line argument. Can you review the Stack Overflow information and see if this satisfies your SSL requirement? Thank you.
https://stackoverflow.com/questions/43595308/set-sslmode-allow-with-pg-dump

Thank you Mike. That covers it.

Thanks @skepticatgit  Did that environment variable work for you by the way? If you didn't test, no worries.  We'll close out the issue for now, but happy to discuss further if needed. Thanks, Jason
#please-close

#please-close


Wait until Postgres initialization done

If Postgres initialization for Icinga is not completed, Icinga API will not work. Wait for these completion in operator startup.

health check is enough for API access. Icingaweb will not work until database initialization.


MKR changes(sms, fixes and change requests)

TODO: adding MSISDN validation

Adding examples for sms

check msisdn with other carrier

sms connectivity readme

wifi tls tutorial

multipart sms check

txt vs bin sms note


Default style not loading

main.js
import 'vuetify/dist/vuetify.min.css'
import 'vuejs-dialog/dist/vuejs-dialog.min.css'

However the style is not applied to dialogs when they are showing up. Any idea why since everything else  is working as expected?

same here..

same here

Can you try loading the script via html tag pending when this is resolved?

Does not work!

Okay, can you help me? want to try reproduce the issue ... how are you using the lib?

All I was doing initially is
import 'vuejs-dialog/vuejs-dialog.min.css'
and the following error wuld pop up:
Module not found: Error: Can't resolve 'vuejs-dialog/vuejs-dialog.min.css'

I see the file in the module, but for some reason it is not detected

I finally got it to work by loading the style sheet but the import fails

Actually, this is what's supposed to be the right path vuejs-dialog/dist/vuejs-dialog.min.css

What do you mean by "the import fails"? is your bundler configured to load css files?

Yes. I did realised and that's how I fixed it. But when I use the same statement as
import "vuejs-dialog/dist/vuejs-dialog.min.css" it fails to import it


Ergonomic Argument Extraction for mruby C functions in manual.rs

Add an Args extractor for Container#intialize in manual.rs.
Restructures extern "C" functions so they live on the Container impl and are declared unsafe.
Add clippy denies to manual.rs.
Add test that Rc is cloned properly in Container#value.

This is work toward GH-7.


Allow using generic type parameters that don't implement Fail

Currently Fail can be derived for types with generic type parameters, as long as they also implement the Fail trait. This means that when I derive fail for MyType<T>, the implementation has type bounds as follows impl<T: Fail> Fail for MyType<T>.
I would like the derived implementation to have custom bounds, so that I can use types that only implement Debug for example.
My use case is that I'm creating an error type for a custom tokio_service::Service that has the generic request and response types. My custom service only allows some requests to be handled, so I want to add an error variant for an unexpected request, something like:
#[derive(Debug, Fail)]
pub enum ServiceError<T: 'static + Debug + Send + Sync> {
    // ...
    #[fail(display = "Unexpected request: {:?}", _0)]
    UnexpectedRequest(T),
    // ...
}

I would suggest that the generic bounds be obtained from the definition of the type to be derived (in the example above, T: 'static + Debug + Send + Sync).

Originally opened on withoutboats/failure_derive, and also opened a PR there.


Lock PHPStan version

PHPStan does not have typical semver versioning and often introduces new checks in "patch" versions (those are not considered patch by author).
This change prevents checks in CI/clean composer installs to fail when new PHPStan version is released (same goal was attempted to be achieved here https://github.com/webonyx/graphql-php/pull/519)

Is there any intention by phpstan to go semvar?
At this point it's not clear yet, it's still a pre-1 version thus semver wouldn't apply either way (at least, that's my understanding).

@mfn there was akward discussion on the topic https://github.com/phpstan/phpstan/issues/2163
There are BC breaks in "patch" versions. I don't complain about it, yet we should be locked against it.

Thanks for the link, appreciated!

Nice, thank you!


All pods related to service not being scraped by service autodiscovery via annotations

Using helm chart 1.4.3. I have had a service being scraped via annotation autodiscovery. Today we noticed that certain metrics were missing. The service has an underlying pod that can grow to up to 8 pods depending on load. If we only have 1 pod all the metrics are scraped. If we have more than one pod behind the service all the metrics are not scraped.
I was under the impression that if I add the k8s.grafana.com/scrape: "true" annotation to a service then alloy would get all the endpoints for that service and scrape them individually. Have I misunderstood how this works? Is there some other reason why this isn't working?
I switched and added the autodiscovery annotation to the deployment template and all the metrics are now being scraped properly via pod autodiscovery. I am so confused. Have I missed something?
Any help would be appreciated. Thanks!

Interesting-- found this stating that discovery.kubernetes.services "discovers a target for each service port for each service." This seems to indicate that not all endpoints (pods) backing a service get set as targets to scrape, but only one.
My assumption was that it was working the same as prometheus.operator.servicemonitors documented here which states that all endpoints for a service are collected and scraped.
This would essentially mean that adding k8s.grafana.com/scrape: "true" annotation to a service with multiple pods backing it will not scrape all the pods for metrics, but only one.
Can someone from Grafana confirm this for me? And if so, is this expected behavior? I would think you would want discovery.kubernetes.services to scrape all underlying endpoints (pods) of a service.

Yeah, currently, services discovery will scrape once per service, pod discovery will scrape per pod.
One of the use cases for the current setup is for services where each pod will respond with the same output, but there might be multiple pods for load balancing or redundancy.
Let me think about this. Perhaps with a special annotation, we can trigger discovery by endpoint vs by service.


chore: remove empty peerDependencies

as npm v3 does not like those.

Probably best to merge this after #14 as that contains fixes for the build which is currently broken (eslint).

Thanks :octocat:


Add "inlineDynamicImports" to rollup config to fix CJS build. Make factory methods basic functions

Add "inlineDynamicImports" to rollup config to fix CJS build.
Make factory methods basic functions.

We're currently processing your upload.  This comment will be updated when the results are available.


updated URL for the  NWChem website

The old NWChem website has been take over by cyber squatters. Please do use the current website https://nwchemgit.github.io

Thanks for the update!


Revert "feat(FX-3109): tag page"

Reverting so that we can fix a type error! I also tried to render on iOS and got a blank screen, but it might be because I was in a hurry and forgot something
Reverts artsy/eigen#5195

Warnings

:warning:

❌ An error occurred while validating your changelog, please make sure you provided a valid changelog.

  Generated by :no_entry_sign: dangerJS against 9d7df38dc801503887d350d4d7af4446c525c5ce


PQL Query() Function to support query options

Overview
To date, query options have been provided as HTTP query parameters. For example,
to return columnAttrs for a given Bitmap result, one would perform a query like:
curl -X POST "http://localhost:10101/index/i/query?columnAttrs=true" -d 'Bitmap(frame="f", row=1)'

There have been several cases where it would useful to expand on the query options
that are currently supported. Short of introducing an entirely different query interface (which
we don't want to undertake right now), it seems we have a couple of options:

Continue to use HTTP query parameters, adding more as needed.
Explore simple modifications to PQL that would allow support for query options.

We decided that it would be reasonable to move forward with option (2), with the
understanding that if it seems like the work required is going to become too
large, we will back off and resort to option (1).
Details
Currently there are only two query options supported by Pilosa: columnAttrs and slices.
The following issues may require additional query options:

#632 - Get RowAttrs without returning bit vector
#633 - Return RowAttrs in TopN queries
#684 - Getting only first 100 bits as output

Proposal
Introduce a wrapper called Query() that wraps existing PQL statements such that the wrapper
function takes one or more arguments: a PQL query (required) and zero or more query options.
Query(Bitmap())
Query(Intersect(Bitmap(), Bitmap()), columnAttrs=true, slices=[1,2])
Query(TopN(Bitmap(), n=10), rowAttrs=true)

One thing to consider is how different result types should be handled. Note how the queries in
the example above have different return types.
Changes should be backward compatible with existing PQL queries, meaning that the following
query would still be valid:
Intersect(Bitmap(), Bitmap())

A successful implementation of this proposal would also remove all existing support for HTTP
query parameters.
Initial Requirements
Explore and report back on the feasibility and difficulty of adding a Query() wrapper
around exising PQL statements.

Nice! I think this is a step in the right direction. Would it make sense to replace Count([...]) with Query([...], output="count") or something similar?

Not that I have a better name, but naming the call Query is a bit strange.

I don't have a strong opinion on naming, but one option would be to use Select() for read-only queries and something like Insert() or Update() for attributes and bit flips.

I like that idea @codysoyland - more generally, could we just do Read(...) and Write(...)?

I like the idea of Read() and Write(). What would you use for a query that does both?

What query does both?

Nothing right now I don't think. This would: https://github.com/pilosa/pilosa/issues/637

it's ok, we'll just add a third: Wread(...)

More seriously, I think if a query is doing any kind of write, it would be considered a Write

Moving to icebox for now until we have a better reason to do this; as it stands it's a lot of work for not much gain.

Now that we've rewritten the parser, this would be pretty straightforward - and probably a good idea to have these options supported outside of HTTP. We have to decide what we want to call it Query, Do, Options... or something separating reads and writes which I'm not sure is necessary at the moment.

@yuce and I discussed the naming - current implementation has Query, but he's going to change it to Opt. Query becomes very confusing to talk about since it's an overloaded word. Any feedback or other suggestions are welcome.

I prefer Options over Opt (but I also prefer column over col and look how that turned out). It seems to me the only argument is that you save parsing four additional characters.

I'm fine with Options


Parsl slows down after hundreds of thousands of jobs

I have a script that requires millions of jobs to be run on parsl and have noticed that after a day or so of running the time per jobs starts increasing. The jobs are independent, nearly identical and randomly ordered so I don't thinks its related to them. I've also changed from Remote_IPP to IPyParallelExecutor locally to IPyParallelExecutor on SLURM and disabled app_cache.
Batching jobs has helped reduce the slowdown so it is likely something internal to parsl causing the issue.
My config is:
rccNode = Config(
    executors=[
        IPyParallelExecutor(
            label='midway_ipp',
            provider=SlurmProvider(
                'broadwl',
                channel=LocalChannel(),
                #launcher=SrunLauncher(),
                init_blocks=1, # 1
                min_blocks=0, # 0
                max_blocks=10, # 10
                nodes_per_block=1, # 4
                tasks_per_node=28, # 12
                walltime="00:30:00",
                overrides="#SBATCH --mem-per-cpu=2000..."
            ),
        ),
    ],
    app_cache=False,
)

This shows jobs run vs time, with a restart in the middle and an OLS. You can see that after the restart the line is much steeper, i.e. jobs are running faster.
I think the simplest way to reproduce the issue would be to run one of the test examples in a loop a million times, but I have not tested it.

This is interesting. It appears that the run-vs-time line is pretty straight before you restart it. So I assume the running speed drops during the first a hundred thousand and keeps the slower speed before a restart? My first guess is the problem of memory-cache locality. Longer execution results in bigger memory usage, and results in the loss of locality of cache, which causes longer memory access time for the master to pull tasks. Can you show me your code and machine environment you were working on? @reidmcy
@yadudoc @kylechard @annawoodard If that is a common case, I think a solution might come up with to solve this slow-down. Probably by refreshing where slows us down and not necessary to keep in memory every hundred thousand tasks.

@MalcolmUnWang the slowdown is nonlinear, it's just not obvious in that picture. Here is a longer time window:

It seems to transition from linear to nonlinear, presumably as slack in the system runs out and loading/unloading tasks becomes the bottleneck.

@MalcolmUnWang I can't show you all my code, but the main loops looks something like:
    running = {}
    try:
        while not done:
            #Only add maxRunning jobs to the queue at once
            numOldJobs = 0 if oldJobs is None else len(oldJobs)
            while len(running) < config.maxRunning:
                #Load next repo
                batch = []
                names = []
                for i in range(config.reposPerBatch): #Currently loading 5
                    try:
                        #Reads a line from a csv file
                        tDat = next(workingTargets)
                    except StopIteration:
                        #If none left just skip adding
                        doneIter = True
                        break
                    else:
                        #Add repo to list of running jobs
                        tDat['downloadDir'] = currentName
                        batch.append(t)
                        names.append(repoName)

            runningNameLists = list(running.keys())
            for names in runningNameLists:
                        if running[names].done():
                            succ, ret_dat, timings = running.pop(names).result()
                            writeResult(repoName, succ, ret_dat)

            #End the loop if all jobs are done and no more can be added
            if doneIter and len(running) < 1:
                done = True

Can you watch the memory usage in top and report if it is continually growing?

The issue isn't just the size in memory, it is that we haven't optimized the search to determine if all of the dependencies are met for a task. I think the entire task tree has to be traversed each time.

This should have been fixed in #600. We can re-open if we see further issues.


Equation 11 confusion

I'm not seeing how this code:
https://github.com/wuga214/NCE_Projected_LRec/blob/master/models/nceplrec.py#L11-L29
Corresponds to this

In particular, this term

is a constant, right? The number of interactions for each item, summed over items... so the total number of interactions, i.e. the number of nonzero entries?
I see you have an unused variable here:
nnz = matrix.nnz

But, in the code, the corresponding part to equation 11 seems to be
values = np.maximum(np.log(rows/np.power(values, root)), 0)

where rows is an integer: the number of users.
What am I missing here?

Hi deklanw,
The root in the code is not exactly the beta in the paper, but something like root = beta/constant
As you have seen, the nominator here is just a constant (the total number of ratings in the training set); another constant could replace it without changing the basic concept.  This is similar to what we do on the SVM models, where the margin could be set as r=1 to reduce the optimization complexity.
We replace it with the number of rows because it could guarantee, log() >=0, so we can sufficiently distinguish rated entry from those of zeros.

Hi deklanw,
The root in the code is not exactly the beta in the paper, but something like root = beta/constant
As you have seen, the nominator here is just a constant (the total number of ratings in the training set); another constant could replace it without changing the basic concept.  This is similar to what we do on the SVM models, where the margin could be set as r=1 to reduce the optimization complexity.
We replace it with the number of rows because it could guarantee, log() >=0, so we can sufficiently distinguish rated entry from those of zeros.

@wuga214 Thanks. That makes sense. A bit tricky, I didn't even notice initially while reading that that value was a constant.
But, I think I got it now.

@wuga214 Thanks. That makes sense. A bit tricky, I didn't even notice initially while reading that that value was a constant.
But, I think I got it now.


Using Mongoose Plugins ?

Is there any way to use the very useful Plugins from mongoose ? Or anything allowing me to re-use a bunch of properties in several models ?
Re-using the a sub schema could be a solution but it does not allow custom pre-validation on on the added properties. I've also tried using typescript mixins, but this method needs to redeclare the variables in the implementing class.
Any ideas about this ?
Thanks in advance.

Thanks for your interest in Typegoose. Plugins are currently not supported however can be easily implemented (which I will try to do in sometime in the near future) as class decorators.
I will change this to enhancement and create a pull request for this when ready.
If I am correct your other question is that is it possible to reuse properties? I haven't tested this but if you create a base class 'A' which extends Typegoose and you extend class 'B' and 'C' from class 'A' then B and C will have A's properties. Is this what you want to do?

Thanks for the plugin annotation works great!
The extension of classes seems not possible at the moment.
The extended classes do not inherit the attributes from the parent class also not the pre decorators.
Don't know if they inherit the static methods.

Hey, awesome work on Typegoose. I'm having issues using the plugin functionality. Tried to implement it using the docs, but can't seem to get it to work. https://stackoverflow.com/questions/45775134/using-plugins-with-typegoose.

Hey @stuartkuentzel, thanks for your interest in Typegoose!
Unfortunately when applying a Mongoose plugin, TypeScript has no information about the plugins "class signature" e.g. which new properties and methods will the plugin add to the schema. This is why the additional typing infos must be provided by hand, as the Typegoose docs states:

If the plugin enhances the schema with additional properties or instance / static methods this typing information should be added manually to the Typegoose class as well.

So in your case you must add authenticate, serializeUser, and deserializeUser to the Typegoose extending User class.
import * as passportLocalMongoose from 'passport-local-mongoose';
import { prop, Typegoose, plugin } from 'typegoose';

@plugin(passportLocalMongoose)
class User extends Typegoose {

    @prop()
    email: string;

    @prop()
    username: string;

    @prop()
    password: string;

    // method return values are not represented here, you'll have to look for the plugin's docs to see what they actually return
    static authenticate();
    static serializeUser();
    static deserializeUser();

}

export const UserModel = new User().getModelForClass(User);

And basically that is it! Hope it helps.
I am going to attach this link to stackoverflow too.

The issue I'm having is, TypeScript doesn't like if I follow with an instanceMethod. If I follow anything after, for example static authenticate(): any;, the following instance method will say "function implementation name must be authenticate". Thanks for your help!

@stuartkuentzel okay i will try to reproduce the problem, meanwhile can you attach what TypeScript / Node / NPM versions are you using?
Thanks!

@szokodiakos Sure, I'm using Node 8.2.1, TypeScript 2.4.2, and NPM 5.3.0.

@szokodiakos any solution to this problem


Excessive buffer flush in wire protocol

For each message peloton receives, peloton processes it and flush the response buffer. To execute a query, JDBC sends over parse, bind, describe, execute and sync (5 messages) all together in one packet, and waits for response. Currently peloton flushes the buffer after processing each of these messages. Instead flushing 5 times, we should batch the 5 responses and just flush once.
@sid1607 @malin1993ml

That's a cool finding. Does that improve our networking performance?

Used no-op benchmark script to stress the networking in peloton, and got the following result (unit: queries / second). Batch flush is the improve version of peloton.
We have significant improvement on local tests, but for remote(cross-machine) test the improvement is not as significant.

No-op query
Timesten
Peloton
Postgres
Peloton (Batch Flush)

Remote
8554
8024
8906
10183

Local
26455
26533
33858
51578

Prep Stmt
Timesten
Peloton
Postgres
Peloton (Batch Flush)

Remote
4827
4476
4245
4632

Local
18158
12073
13896
17774

** No op query = “;”
** Prep stmt = “select * from A where id = ?”. Table A is empty.
** Remote = JDBC on dev2, database on dev1
** Local = Both JDBC & database on dev1

This is cool, I can run timesten ycsb+tpcc right after this is merged


List of useful Git Hook and .gitignore files

I thinks these ones would make a great addition to the list

https://github.com/pre-commit/pre-commit
https://github.com/github/gitignore

Given the fact that by then a number of Git related items are in the list maybe it makes sense to move Git (and possibly GitHub) to a separate section?

To be clear, this repo is a list of other lists of some kind of resource. None of the links above fits that category.


Use resolver.URIResolver in KafkaChannel

Describe the bug
KafkaChannel uses only the URI field of the DeadLetterSink: https://github.com/knative/eventing-contrib/blob/62ccf3743fbe0f1d0f8339e9636a85592880b7e9/kafka/channel/pkg/dispatcher/dispatcher.go#L147
Expected behavior
KafkaChannel uses resolver.URIResolver for resolving DeadLetterSink.
To Reproduce
Use Ref instead of URI in DeadLetterSink
Knative release version
<= devel

/assign

We don't need this, because it's already done by the subscription controller in eventing: https://github.com/knative/eventing/blob/master/pkg/reconciler/subscription/subscription.go#L558

/close


Language-specific guides request c#. Net core

Problem description

Problem location

I saw a problem on the following URL:

I couldn't find the information I wanted. I expected to find it near the following URL:

Other:

Project version(s) affected

Suggestions for a fix

Thanks for requesting; let me close this as duplicate of https://github.com/docker/docker.github.io/issues/12697


cannot exclude categories

The CI cannot run some of the test, so usually I exclude them in the CI.
However I don't found a way to do it with dotnet test.

Are you marking the tests as explicit? If so then, it should work on CI and command line in general but not when running from VS: https://github.com/nunit/dotnet-test-nunit/issues/25.

No, I only set a category to them. I will use the explicit attribute instead.
Thanks.

You can also add the --where command line option from the NUnit console to run/exclude specific tests or categories. See the NUnit console docs for info. Unfortunately dotnet test does not show the command line options of the runner even though we provide them :-1:

We can send command line options to process via argument separator --; i.e.
dotnet test path/to/test/dir -- --where ..


'Access token' and even 'user ID and password' can be confused with turnserver.conf credentials

@reivilibre aha, I see! I mistook "user id and password" and "access token" to be based on the nomenclature in turnserver.conf (i.e. the static-auth-secret and "user=username1:password1").
Originally posted by @rht in https://github.com/matrix-org/synapse/issues/9135#issuecomment-763509027

For discoverability, there could be a hover link over the "Access Token" to a page/hover-text that tells how to get the access token (e.g. "(in Riot/Web, you can look at Settings → Help & About → Access token: >click to reveal>").

For discoverability, there could be a hover link over the "Access Token" to a page/hover-text that tells how to get the access token (e.g. "(in Riot/Web, you can look at Settings → Help & About → Access token: >click to reveal>").

A mode in which you specify the turn-server URL(s) and the shared secret might be useful anyway.

A mode in which you specify the turn-server URL(s) and the shared secret might be useful anyway.


Can't connect to gRPC with external IP

Background
I want to access gRPC with external ip.
here is my lnd.conf
[Application Options]
externalip=87.227.XXX.XX
alias=XXXXXXXXX
color=#3399FF
[Bitcoin]
bitcoin.active=1
bitcoin.mainnet=1
bitcoin.node=bitcoind
[Bitcoind]
bitcoind.rpcuser=XXXXXXXX
bitcoind.rpcpass=XXXXXXXXXX
bitcoind.zmqpubrawblock=XXXXXXXX
bitcoind.zmqpubrawtx=XXXXXXXX
I had tried with the localhost:10009 and it's working but I need to figure out how and on which port to make it work with the external IP 87.227.XXX.XX .
I don't know if I need to explicitly to config this option, but I didn't find any doc.
Testing on my LAN .
Here are my LND LISTEN TCP connections.

Your environment
OSX

version of lnd : 0.5.0-beta
which operating system : OSX
version of bitcoind : v<IP_ADDRESS>

Actual behaviour
Can't use gRPC with external IP .

And you'll need to regenerate your by deleting them and restarting lnd if you set a tlsextraip

Closing based on the responses above.


Create IndyVdrAnonCredsRegistry without revocation

Create an IndyVdrAnonCredsRegistry that implements the AnonCredsRegistry interface from the @aries-framework/anoncreds package using the indy vdr library.

Implement the AnonCredsRegistry interface methods from the anoncreds package.

read/write schema

read/write credential definition

return not implemented error for revocation status list and revocation registry definition

add tests

should be added to the @aries-framework/indy-vdr package

Specification: https://hyperledger.github.io/anoncreds-spec/

https://hyperledger.github.io/indy-did-method/#other-indy-ledger-object-identifiers

Can use the IndySdkAnonCredsRegistry as an example, but must be updated to use the IndyVdrPoolService and IndyVdrPool.
We may need to transform the structure of some objects. As the AnonCreds interfaces are different than what is stored on the indy ledger. However, the same transformations are present in the IndySdkAnonCredsRegistry class, so should make sure it is correct, but we can mostly copy the transformations from the existing implementaiton.

Working on this!


Don't use dev-master

Why is the dev-master of flarum api is required in the composer file? It causes issue when getting installed to another framework.

Got it working, thanks for this plugin.

Hi,
It is required the dev master because I need some methods that aren't in the 1.x beta.
If you explain your issue and your solution you may help others that have the same problems


Fix issues

This is mainly updating filters and stationxml

Updated timeseries.filters to work as other metadata classes
Updated timeseries.experiment to pull through all types of filters encountered so far
Updated timeseries.stationxml to work with all filters encountered so far
Removed redundant files
Updated logging and added concurrent-log-handler package to setup to remove error of using multiple python processes that use mt_metadata
Updated tests to make sure all the updates work

Codecov Report

Merging #39 (e22aa8b) into main (7242fb8) will increase coverage by 4.27%.
The diff coverage is 73.82%.

@@            Coverage Diff             @@
##             main      #39      +/-   ##
==========================================
+ Coverage   68.66%   72.93%   +4.27%
==========================================
  Files         154      155       +1
  Lines        7875     7774     -101
==========================================
+ Hits         5407     5670     +263
+ Misses       2468     2104     -364

Impacted Files
Coverage Δ

mt_metadata/timeseries/channel.py
88.52% <ø> (ø)

...ta/timeseries/stationxml/xml_station_mt_station.py
77.37% <ø> (ø)

setup.py
0.00% <0.00%> (ø)

tests/__init__.py
100.00% <ø> (ø)

tests/timeseries/stationxml/test_station.py
99.30% <ø> (ø)

mt_metadata/timeseries/experiment.py
61.17% <6.66%> (-4.93%)
:arrow_down:

mt_metadata/timeseries/stationxml/fdsn_tools.py
75.00% <25.00%> (-5.00%)
:arrow_down:

...meseries/stationxml/xml_inventory_mt_experiment.py
74.04% <33.33%> (ø)

mt_metadata/utils/mt_logger.py
72.72% <33.33%> (-22.63%)
:arrow_down:

...data/timeseries/filters/channel_response_filter.py
65.11% <38.88%> (-4.06%)
:arrow_down:

... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bf03631...e22aa8b. Read the comment docs.


Quick start for loading variants data

@mbookman please let me know what feedback you have on this.

The idea is that it is a "quick start" for loading data.
It is not as detailed as the material in the "Storing Genomic Data" on https://cloud.google.com/genomics but it gives you the end-to-end steps which are currently not covered anywhere in our docs.
The figure will be updated at a later time with proper icons instead of boxes.

I also haven't decided yet where it should live.  Some options are:

http://google-genomics.readthedocs.org/ (this is where it would currently go)
or perhaps consider it a codelab (even though it has no code) and put it under https://github.com/googlegenomics/codelabs

@gregmcinnes can you please give this a quick read and let me know if you have any feedback?  Thanks!

@deflaux looks good to me!  Very helpful.

Hi Nicole,
Good stuff.  Just one comment - in a number of places like:
THE_NEW_EXPORT_JOB_ID This was returned in the output of the prior command.
I would like to see the identifier separated from the description, by a
colon:
THE_NEW_EXPORT_JOB_ID: This was returned in the output of the prior command.
-Matt
On Wed, Mar 4, 2015 at 11:11 AM Greg<EMAIL_ADDRESS>wrote:

@deflaux https://github.com/deflaux looks good to me! Very helpful.
—
Reply to this email directly or view it on GitHub
https://github.com/googlegenomics/start-here/pull/8#issuecomment-77224259
.


updated stylelint-order to the latest version

Updated the stylelint-order to the latest version. The author has added the stylelint as a peer dependency, so the packages tree should look nicer now.
*Not sure if it was correct to update the npm package version right away. If you don't like it I'll roll it back. *

Hi @stukalin thanks for the PR. Before I merge this, please remove the version bump, I will do that on release.

@danilobuerger I knew it was too quick! Will do shortly

Thanks! What version of npm are you using? I am getting different results on the package-lock.json

@danilobuerger I used the node v10.8.0 with npm v6.2.0. Which one is yours?

The same, I wonder why I am getting a different result. On npm install on your patch, the diff is:
diff --git a/package-lock.json b/package-lock.json
index b38e26b..ab468dd 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -88,7 +88,7 @@
     "arr-flatten": {
       "version": "1.1.0",
       "resolved": "https://registry.npmjs.org/arr-flatten/-/arr-flatten-1.1.0.t
-      "integrity": "sha1-NgSLv/TntH4TZkQxbJlmnqWukfE=",
+      "integrity": "sha512-L3hKV5R/p5o81R7O02IGnwpDmkp6E982XhtbuwSe3O4qOtMMMtod
       "dev": true
     },
     "array-find-index": {
@@ -287,7 +287,7 @@
     "circular-json": {
       "version": "0.3.3",
       "resolved": "https://registry.npmjs.org/circular-json/-/circular-json-0.3
-      "integrity": "sha1-gVyZ6oT2gJUp0vRXkb34JxE1LWY=",
+      "integrity": "sha512-UZK3NBx2Mca+b5LsG7bY183pHWt5Y1xts4P3Pz7ENTwGVnJOUWbR
       "dev": true
     },
     "cli-cursor": {
@@ -1135,7 +1135,7 @@
           "integrity": "sha1-qEeQIusaw2iocTibY1JixQXuNo8=",
           "dev": true,
           "requires": {
-            "ansi-regex": "3.0.0"
+            "ansi-regex": "^3.0.0"
           }
         },
         "supports-color": {
@@ -1883,7 +1883,7 @@
           "integrity": "sha1-ZawFBLOVQXHYpklGsq48u4pfVPY=",
           "dev": true,
           "requires": {
-            "has-flag": "1.0.0"
+            "has-flag": "^1.0.0"
           }
         }
       }

Strange. Can we somehow stuck to the version which makes less changes with the master? Or to yours just to be on the safe side

Ok, so I made a new commit and set you as author, closing this. Thanks!

Published as 2.0.2


:bug: : errors in ErrXXX format

What kind of change does this PR introduce?
fix linting expections around errors
FIX Took care of the //nolint:errname directives while we're doing breaking changes.

[x] PR title follows the guidelines defined in our pull request documentation

What is the current behavior?
uses directive to skip lint
What is the new behavior (if this is a feature change)?**
does not skip lint, fixed the errors

[x] Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes
Fixes #4033
Special notes for your reviewer
From OSSNA workshop
Does this PR introduce a user-facing change?
NONE

@spencerschrock 👀


[6.0] Add --enable-code-coverage to swift build

Explanation: Add --enable-code-coverage to swift build. This flag allows enabling code coverage when splitting a test run between swift build and swift test --skip-build.
Scope: New option for the swift build command.
Original PR: #7508 (partial), #7518
Risk: Low. Behaviour is identical for swift build without the new flag, and the new flag must be explicitly specified.
Testing: Ran swift build unit tests at desk and in CI.
Reviewer: @bnbarham, @MaxDesiatov, @stmontgomery

@swift-ci please test


Fix several type annotations for mypy 0.4.6 compatibility.

A few functions had arguments removed without having their type
annotations updated accordingly. As a result mypy version 0.4.6
thinks that the first type in the annotation is supposed to be
the type of self, a new mypy feature which we are not intending
to use here.

Automated message from Dropbox CLA bot
@rwbarton, it looks like you've already signed the Dropbox CLA. Thanks!

You can also just add a commit upgrading our mypy in requirements/mypy.txt.

There's another issue with tools/deprecated/iframe-bot/show-last-messages.py, but I don't understand why it was supposedly passing before.

Okay, I provisioned and ran mypy from inside the VM and that file does pass with mypy 0.4.5, so I can investigate what's going on here.

I think that's actually a mypy caching bug; try moving aside var/mypy-cache...

Merged, thanks @rwbarton!

Okay here is the situation. There is a type error in tools/deprecated/iframe-bot/show-last-messages, but this bug https://github.com/python/mypy/issues/2523 in mypy 0.4.5 masks it. However, that bug was fixed in 0.4.6, so new mypy correctly reports an error. But the type error is due to incomplete stubs for codecs (https://github.com/python/typeshed/issues/732), so I can't just fix it.
Should I fix it with a cast and then bump the mypy version? Or just leave this as is for now.

What we usually do when working around this sort of issue is add type: ignore to deal with this sort of situation, with a link to the mypy/typeshed bug ID that caused us to have to add it, so it's easy to clean things up later when the bug is fixed.


Linear evaluation

Thanks for your excellent work! But i have trouble in linear evaluation.
I get the model from Model Zoo (BYOL (@XiaohangZhan) | selfsup/byol/r50_bs4096_ep200.py (@xieenze) | byol_r50-e3b0c442.pth), then I  use  it to do linear evaluation with the config-configs/benchmarks/linear_classification/imagenet/r50_last.py  . The result is very bad, acc is very low and the loss does not converge.
INFO - head0_top1: 0.100
2021-01-06 10:17:54,353 - openselfsup - INFO - head0_top5: 0.530
2021-01-06 10:19:28,189 - openselfsup - INFO - Epoch [1][50/10010]      lr: 3.000e+01, eta: 21 days, 16:32:18, time: 1.872, data_time: 1.130, memory: 3223, loss: 6.9107, acc: 0.0781
2021-01-06 10:20:23,091 - openselfsup - INFO - Epoch [1][100/10010]     lr: 3.000e+01, eta: 17 days, 4:53:54, time: 1.098, data_time: 0.005, memory: 3223, loss: 6.9106, acc: 0.0938
2021-01-06 10:21:22,120 - openselfsup - INFO - Epoch [1][150/10010]     lr: 3.000e+01, eta: 16 days, 0:39:29, time: 1.181, data_time: 0.212, memory: 3223, loss: 6.9093, acc: 0.1875
2021-01-06 10:22:06,845 - openselfsup - INFO - Epoch [1][200/10010]     lr: 3.000e+01, eta: 14 days, 14:38:47, time: 0.894, data_time: 0.044, memory: 3223, loss: 6.9090, acc: 0.1406
2021-01-06 10:22:49,104 - openselfsup - INFO - Epoch [1][250/10010]     lr: 3.000e+01, eta: 13 days, 15:29:32, time: 0.845, data_time: 0.012, memory: 3223, loss: 6.9092, acc: 0.0625
2021-01-06 10:23:24,521 - openselfsup - INFO - Epoch [1][300/10010]     lr: 3.000e+01, eta: 12 days, 17:42:46, time: 0.708, data_time: 0.194, memory: 3223, loss: 6.9103, acc: 0.0781
2021-01-06 10:24:01,836 - openselfsup - INFO - Epoch [1][350/10010]     lr: 3.000e+01, eta: 12 days, 3:39:37, time: 0.746, data_time: 0.080, memory: 3223, loss: 6.9107, acc: 0.0938
2021-01-06 10:24:39,800 - openselfsup - INFO - Epoch [1][400/10010]     lr: 3.000e+01, eta: 11 days, 17:34:12, time: 0.759, data_time: 0.553, memory: 3223, loss: 6.9110, acc: 0.0781
2021-01-06 10:25:42,831 - openselfsup - INFO - Epoch [1][450/10010]     lr: 3.000e+01, eta: 12 days, 1:12:04, time: 1.261, data_time: 0.118, memory: 3223, loss: 6.9087, acc: 0.1250
2021-01-06 10:26:34,674 - openselfsup - INFO - Epoch [1][500/10010]     lr: 3.000e+01, eta: 12 days, 1:05:01, time: 1.037, data_time: 0.009, memory: 3223, loss: 6.9097, acc: 0.0312
2021-01-06 10:27:19,907 - openselfsup - INFO - Epoch [1][550/10010]     lr: 3.000e+01, eta: 11 days, 21:38:43, time: 0.905, data_time: 0.071, memory: 3223, loss: 6.9083, acc: 0.1875
2021-01-06 10:27:57,860 - openselfsup - INFO - Epoch [1][600/10010]     lr: 3.000e+01, eta: 11 days, 15:24:23, time: 0.759, data_time: 0.006, memory: 3223, loss: 6.9091, acc: 0.0469
2021-01-06 10:28:35,339 - openselfsup - INFO - Epoch [1][650/10010]     lr: 3.000e+01, eta: 11 days, 9:55:22, time: 0.750, data_time: 0.240, memory: 3223, loss: 6.9095, acc: 0.0781
2021-01-06 10:29:15,188 - openselfsup - INFO - Epoch [1][700/10010]     lr: 3.000e+01, eta: 11 days, 6:09:42, time: 0.797, data_time: 0.010, memory: 3223, loss: 6.9095, acc: 0.0000
2021-01-06 10:29:50,669 - openselfsup - INFO - Epoch [1][750/10010]     lr: 3.000e+01, eta: 11 days, 1:16:58, time: 0.710, data_time: 0.490, memory: 3223, loss: 6.9088, acc: 0.1094
2021-01-06 10:30:55,669 - openselfsup - INFO - Epoch [1][800/10010]     lr: 3.000e+01, eta: 11 days, 7:15:50, time: 1.300, data_time: 1.190, memory: 3223, loss: 6.9089, acc: 0.1094
2021-01-06 10:31:42,625 - openselfsup - INFO - Epoch [1][850/10010]     lr: 3.000e+01, eta: 11 days, 6:38:31, time: 0.939, data_time: 0.574, memory: 3223, loss: 6.9096, acc: 0.0625
2021-01-06 10:32:24,059 - openselfsup - INFO - Epoch [1][900/10010]     lr: 3.000e+01, eta: 11 days, 4:22:58, time: 0.829, data_time: 0.685, memory: 3223, loss: 6.9096, acc: 0.1094
2021-01-06 10:33:04,725 - openselfsup - INFO - Epoch [1][950/10010]     lr: 3.000e+01, eta: 11 days, 2:08:10, time: 0.813, data_time: 0.138, memory: 3223, loss: 6.9096, acc: 0.1250
2021-01-06 10:33:43,538 - openselfsup - INFO - Epoch [1][1000/10010]    lr: 3.000e+01, eta: 10 days, 23:35:52, time: 0.776, data_time: 0.016, memory: 3223, loss: 6.9105, acc: 0.0938
2021-01-06 10:34:18,924 - openselfsup - INFO - Epoch [1][1050/10010]    lr: 3.000e+01, eta: 10 days, 20:23:37, time: 0.708, data_time: 0.107, memory: 3223, loss: 6.9107, acc: 0.1250
2021-01-06 10:34:53,551 - openselfsup - INFO - Epoch [1][1100/10010]    lr: 3.000e+01, eta: 10 days, 17:17:18, time: 0.693, data_time: 0.028, memory: 3223, loss: 6.9074, acc: 0.1250
2021-01-06 10:36:03,863 - openselfsup - INFO - Epoch [1][1150/10010]    lr: 3.000e+01, eta: 10 days, 23:04:14, time: 1.406, data_time: 0.225, memory: 3223, loss: 6.9089, acc: 0.0781
2021-01-06 10:36:50,300 - openselfsup - INFO - Epoch [1][1200/10010]    lr: 3.000e+01, eta: 10 days, 22:50:38, time: 0.929, data_time: 0.495, memory: 3223, loss: 6.9094, acc: 0.0781
2021-01-06 10:37:32,179 - openselfsup - INFO - Epoch [1][1250/10010]    lr: 3.000e+01, eta: 10 days, 21:37:18, time: 0.838, data_time: 0.497, memory: 3223, loss: 6.9096, acc: 0.0781
2021-01-06 10:38:13,449 - openselfsup - INFO - Epoch [1][1300/10010]    lr: 3.000e+01, eta: 10 days, 20:21:44, time: 0.825, data_time: 0.171, memory: 3223, loss: 6.9097, acc: 0.1250
2021-01-06 10:38:49,444 - openselfsup - INFO - Epoch [1][1350/10010]    lr: 3.000e+01, eta: 10 days, 18:06:38, time: 0.720, data_time: 0.458, memory: 3223, loss: 6.9100, acc: 0.1719
2021-01-06 10:39:23,363 - openselfsup - INFO - Epoch [1][1400/10010]    lr: 3.000e+01, eta: 10 days, 15:36:25, time: 0.678, data_time: 0.192, memory: 3223, loss: 6.9101, acc: 0.0781
2021-01-06 10:40:13,955 - openselfsup - INFO - Epoch [1][1450/10010]    lr: 3.000e+01, eta: 10 days, 16:28:05, time: 1.012, data_time: 0.242, memory: 3223, loss: 6.9092, acc: 0.1562
2021-01-06 10:41:16,828 - openselfsup - INFO - Epoch [1][1500/10010]    lr: 3.000e+01, eta: 10 days, 19:32:39, time: 1.257, data_time: 0.360, memory: 3223, loss: 6.9111, acc: 0.0625
2021-01-06 10:42:02,019 - openselfsup - INFO - Epoch [1][1550/10010]    lr: 3.000e+01, eta: 10 days, 19:15:12, time: 0.904, data_time: 0.324, memory: 3223, loss: 6.9088, acc: 0.0625
2021-01-06 10:42:43,456 - openselfsup - INFO - Epoch [1][1600/10010]    lr: 3.000e+01, eta: 10 days, 18:19:43, time: 0.829, data_time: 0.141, memory: 3223, loss: 6.9093, acc: 0.0312
2021-01-06 10:43:22,875 - openselfsup - INFO - Epoch [1][1650/10010]    lr: 3.000e+01, eta: 10 days, 17:07:12, time: 0.788, data_time: 0.326, memory: 3223, loss: 6.9098, acc: 0.0469
2021-01-06 10:43:58,435 - openselfsup - INFO - Epoch [1][1700/10010]    lr: 3.000e+01, eta: 10 days, 15:21:05, time: 0.711, data_time: 0.283, memory: 3223, loss: 6.9104, acc: 0.1094
2021-01-06 10:44:35,390 - openselfsup - INFO - Epoch [1][1750/10010]    lr: 3.000e+01, eta: 10 days, 13:54:17, time: 0.739, data_time: 0.651, memory: 3223, loss: 6.9094,

Please check follows:

Did you pull the latest code?
Did you modify the script: benchmarks/dist_train_linear.sh or the config: configs/benchmarks/linear_classification/imagenet/r50_last.py?
Could you provide your command to launch it?

Please check follows:

Did you pull the latest code?
Did you modify the script: benchmarks/dist_train_linear.sh or the config: configs/benchmarks/linear_classification/imagenet/r50_last.py?
Could you provide your command to launch it?

No, not the latest code,  I pull the code in 2020.11
2.I use 4 GPUs, so I change the GPUS=4 in  benchmarks/dist_train_linear.sh and imgs_per_gpu=64 in configs/benchmarks/linear_classification/imagenet/r50_last.py
3.bash benchmarks/dist_train_linear.sh  configs/benchmarks/linear_classification/imagenet/r50_last.py pth/byol_r50.pth
As  I put the model in pth/ and rename it as byol_r50.pth.

No, not the latest code,  I pull the code in 2020.11
2.I use 4 GPUs, so I change the GPUS=4 in  benchmarks/dist_train_linear.sh and imgs_per_gpu=64 in configs/benchmarks/linear_classification/imagenet/r50_last.py
3.bash benchmarks/dist_train_linear.sh  configs/benchmarks/linear_classification/imagenet/r50_last.py pth/byol_r50.pth
As  I put the model in pth/ and rename it as byol_r50.pth.

Please re-clone the latest code (do not merge your modifications) and try again. If not solved, please provide the full log.

Please re-clone the latest code (do not merge your modifications) and try again. If not solved, please provide the full log.

Thank you very much for your reply ! I pull the latest code and rerun then I get the some problem as issue #61. Thanks again！

Thank you very much for your reply ! I pull the latest code and rerun then I get the some problem as issue #61. Thanks again！

So is the problem solved?

So is the problem solved?

When I use the latest code, the log:
2021-01-06 12:12:46,800 - openselfsup - INFO - head0_top1: 0.100
2021-01-06 12:12:46,825 - openselfsup - INFO - head0_top5: 0.530
2021-01-06 12:15:17,168 - openselfsup - INFO - Epoch [1][50/5005]	lr: 3.000e+01, eta: 17 days, 9:19:53, time: 3.002, data_time: 1.462, memory: 3223, loss: 68.5507, acc: 14.2891
2021-01-06 12:17:00,237 - openselfsup - INFO - Epoch [1][100/5005]	lr: 3.000e+01, eta: 14 days, 15:54:36, time: 2.061, data_time: 0.113, memory: 3223, loss: 100.2174, acc: 26.2031
I use code version = '0.3.0+df3689d', the log:
2021-01-06 13:16:01,694 - openselfsup - INFO - head0_top1: 0.100
2021-01-06 13:16:01,696 - openselfsup - INFO - head0_top5: 0.530
2021-01-06 13:18:21,756 - openselfsup - INFO - Epoch [1][50/5005]	lr: 3.000e+01, eta: 16 days, 4:52:13, time: 2.797, data_time: 1.248, memory: 3223, loss: 6.9106, acc: 0.0625
2021-01-06 13:19:37,894 - openselfsup - INFO - Epoch [1][100/5005]	lr: 3.000e+01, eta: 12 days, 12:14:56, time: 1.523, data_time: 0.065, memory: 3223, loss: 6.9092, acc: 0.1328
2021-01-06 13:21:26,304 - openselfsup - INFO - Epoch [1][150/5005]	lr: 3.000e+01, eta: 12 days, 12:35:42, time: 2.168, data_time: 1.578, memory: 3223, loss: 6.9099, acc: 0.0625
But between the two code,  the change that can influent linear evaluation only  is  support data prefetching on the linear classification task  .  But this should not affect the training process, because the default value is false. So I am still in doubt.  The full log as follow:
2021-01-06 13:12:22,073 - openselfsup - INFO - Environment info:
sys.platform: linux
Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.105
GPU 0,1,2,3: GeForce RTX 2080 Ti
GCC: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
PyTorch: 1.7.1+cu92
PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 9.2
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
CuDNN 7.6.3
Magma 2.5.2
Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.8.2+cu92
OpenCV: 4.4.0
MMCV: 1.0.3
OpenSelfSup: 0.3.0+df3689d
2021-01-06 13:12:22,073 - openselfsup - INFO - Distributed training: True
2021-01-06 13:12:22,073 - openselfsup - INFO - Config:
/data1/tangtao/OpenSelfSup/configs/base.py
train_cfg = {}
test_cfg = {}
optimizer_config = dict()  # grad_clip, coalesce, bucket_size_mb
yapf:disable
log_config = dict(
interval=50,
hooks=[
dict(type='TextLoggerHook'),
dict(type='TensorboardLoggerHook')
])
yapf:enable
runtime settings
dist_params = dict(backend='nccl')
cudnn_benchmark = True
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
/data1/tangtao/OpenSelfSup/configs/benchmarks/linear_classification/imagenet/r50_last.py
base = '../../../base.py'
model settings
model = dict(
type='Classification',
pretrained=None,
with_sobel=False,
backbone=dict(
type='ResNet',
depth=50,
in_channels=3,
out_indices=[4],  # 0: conv-1, x: stage-x
norm_cfg=dict(type='BN'),
frozen_stages=4),
head=dict(
type='ClsHead', with_avg_pool=True, in_channels=2048,
num_classes=1000))
dataset settings
data_source_cfg = dict(
type='ImageNet',
memcached=True,
mclient_path='/mnt/lustre/share/memcached_client')
data_train_list = 'data/imagenet/meta/train_labeled.txt'
data_train_root = 'data/imagenet/train'
data_test_list = 'data/imagenet/meta/val_labeled.txt'
data_test_root = 'data/imagenet/val'
dataset_type = 'ClassificationDataset'
img_norm_cfg = dict(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
train_pipeline = [
dict(type='RandomResizedCrop', size=224),
dict(type='RandomHorizontalFlip'),
dict(type='ToTensor'),
dict(type='Normalize', **img_norm_cfg),
]
test_pipeline = [
dict(type='Resize', size=256),
dict(type='CenterCrop', size=224),
dict(type='ToTensor'),
dict(type='Normalize', **img_norm_cfg),
]
data = dict(
imgs_per_gpu=64,  # total 32*8=256, 8GPU linear cls
workers_per_gpu=5,
train=dict(
type=dataset_type,
data_source=dict(
list_file=data_train_list, root=data_train_root,
**data_source_cfg),
pipeline=train_pipeline),
val=dict(
type=dataset_type,
data_source=dict(
list_file=data_test_list, root=data_test_root, **data_source_cfg),
pipeline=test_pipeline))
additional hooks
custom_hooks = [
dict(
type='ValidateHook',
dataset=data['val'],
initial=True,
interval=1,
imgs_per_gpu=128,
workers_per_gpu=4,
eval_param=dict(topk=(1, 5)))
]
optimizer
optimizer = dict(type='SGD', lr=30., momentum=0.9, weight_decay=0.)
learning policy
lr_config = dict(policy='step', step=[60, 80])
checkpoint_config = dict(interval=10)
runtime settings
total_epochs = 100
2021-01-06 13:12:22,073 - openselfsup - INFO - Set random seed to 0, deterministic: False
2021-01-06 13:12:22,334 - openselfsup - INFO - load model from: pth/byol_r50.pth
2021-01-06 13:12:30,498 - openselfsup - INFO - Start running, host: tangtao@lm01, work_dir: /data1/tangtao/OpenSelfSup/work_dirs/benchmarks/linear_classification/imagenet/r50_last/byol_r50.pth/train_20210106_131221
2021-01-06 13:12:30,498 - openselfsup - INFO - workflow: [('train', 1)], max: 100 epochs
2021-01-06 13:16:01,694 - openselfsup - INFO - head0_top1: 0.100
2021-01-06 13:16:01,696 - openselfsup - INFO - head0_top5: 0.530
2021-01-06 13:18:21,756 - openselfsup - INFO - Epoch [1][50/5005]	lr: 3.000e+01, eta: 16 days, 4:52:13, time: 2.797, data_time: 1.248, memory: 3223, loss: 6.9106, acc: 0.0625
2021-01-06 13:19:37,894 - openselfsup - INFO - Epoch [1][100/5005]	lr: 3.000e+01, eta: 12 days, 12:14:56, time: 1.523, data_time: 0.065, memory: 3223, loss: 6.9092, acc: 0.1328
2021-01-06 13:21:26,304 - openselfsup - INFO - Epoch [1][150/5005]	lr: 3.000e+01, eta: 12 days, 12:35:42, time: 2.168, data_time: 1.578, memory: 3223, loss: 6.9099, acc: 0.0625

When I use the latest code, the log:
2021-01-06 12:12:46,800 - openselfsup - INFO - head0_top1: 0.100
2021-01-06 12:12:46,825 - openselfsup - INFO - head0_top5: 0.530
2021-01-06 12:15:17,168 - openselfsup - INFO - Epoch [1][50/5005]	lr: 3.000e+01, eta: 17 days, 9:19:53, time: 3.002, data_time: 1.462, memory: 3223, loss: 68.5507, acc: 14.2891
2021-01-06 12:17:00,237 - openselfsup - INFO - Epoch [1][100/5005]	lr: 3.000e+01, eta: 14 days, 15:54:36, time: 2.061, data_time: 0.113, memory: 3223, loss: 100.2174, acc: 26.2031
I use code version = '0.3.0+df3689d', the log:
2021-01-06 13:16:01,694 - openselfsup - INFO - head0_top1: 0.100
2021-01-06 13:16:01,696 - openselfsup - INFO - head0_top5: 0.530
2021-01-06 13:18:21,756 - openselfsup - INFO - Epoch [1][50/5005]	lr: 3.000e+01, eta: 16 days, 4:52:13, time: 2.797, data_time: 1.248, memory: 3223, loss: 6.9106, acc: 0.0625
2021-01-06 13:19:37,894 - openselfsup - INFO - Epoch [1][100/5005]	lr: 3.000e+01, eta: 12 days, 12:14:56, time: 1.523, data_time: 0.065, memory: 3223, loss: 6.9092, acc: 0.1328
2021-01-06 13:21:26,304 - openselfsup - INFO - Epoch [1][150/5005]	lr: 3.000e+01, eta: 12 days, 12:35:42, time: 2.168, data_time: 1.578, memory: 3223, loss: 6.9099, acc: 0.0625
But between the two code,  the change that can influent linear evaluation only  is  support data prefetching on the linear classification task  .  But this should not affect the training process, because the default value is false. So I am still in doubt.  The full log as follow:
2021-01-06 13:12:22,073 - openselfsup - INFO - Environment info:
sys.platform: linux
Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.105
GPU 0,1,2,3: GeForce RTX 2080 Ti
GCC: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
PyTorch: 1.7.1+cu92
PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 9.2
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
CuDNN 7.6.3
Magma 2.5.2
Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.8.2+cu92
OpenCV: 4.4.0
MMCV: 1.0.3
OpenSelfSup: 0.3.0+df3689d
2021-01-06 13:12:22,073 - openselfsup - INFO - Distributed training: True
2021-01-06 13:12:22,073 - openselfsup - INFO - Config:
/data1/tangtao/OpenSelfSup/configs/base.py
train_cfg = {}
test_cfg = {}
optimizer_config = dict()  # grad_clip, coalesce, bucket_size_mb
yapf:disable
log_config = dict(
interval=50,
hooks=[
dict(type='TextLoggerHook'),
dict(type='TensorboardLoggerHook')
])
yapf:enable
runtime settings
dist_params = dict(backend='nccl')
cudnn_benchmark = True
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
/data1/tangtao/OpenSelfSup/configs/benchmarks/linear_classification/imagenet/r50_last.py
base = '../../../base.py'
model settings
model = dict(
type='Classification',
pretrained=None,
with_sobel=False,
backbone=dict(
type='ResNet',
depth=50,
in_channels=3,
out_indices=[4],  # 0: conv-1, x: stage-x
norm_cfg=dict(type='BN'),
frozen_stages=4),
head=dict(
type='ClsHead', with_avg_pool=True, in_channels=2048,
num_classes=1000))
dataset settings
data_source_cfg = dict(
type='ImageNet',
memcached=True,
mclient_path='/mnt/lustre/share/memcached_client')
data_train_list = 'data/imagenet/meta/train_labeled.txt'
data_train_root = 'data/imagenet/train'
data_test_list = 'data/imagenet/meta/val_labeled.txt'
data_test_root = 'data/imagenet/val'
dataset_type = 'ClassificationDataset'
img_norm_cfg = dict(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
train_pipeline = [
dict(type='RandomResizedCrop', size=224),
dict(type='RandomHorizontalFlip'),
dict(type='ToTensor'),
dict(type='Normalize', **img_norm_cfg),
]
test_pipeline = [
dict(type='Resize', size=256),
dict(type='CenterCrop', size=224),
dict(type='ToTensor'),
dict(type='Normalize', **img_norm_cfg),
]
data = dict(
imgs_per_gpu=64,  # total 32*8=256, 8GPU linear cls
workers_per_gpu=5,
train=dict(
type=dataset_type,
data_source=dict(
list_file=data_train_list, root=data_train_root,
**data_source_cfg),
pipeline=train_pipeline),
val=dict(
type=dataset_type,
data_source=dict(
list_file=data_test_list, root=data_test_root, **data_source_cfg),
pipeline=test_pipeline))
additional hooks
custom_hooks = [
dict(
type='ValidateHook',
dataset=data['val'],
initial=True,
interval=1,
imgs_per_gpu=128,
workers_per_gpu=4,
eval_param=dict(topk=(1, 5)))
]
optimizer
optimizer = dict(type='SGD', lr=30., momentum=0.9, weight_decay=0.)
learning policy
lr_config = dict(policy='step', step=[60, 80])
checkpoint_config = dict(interval=10)
runtime settings
total_epochs = 100
2021-01-06 13:12:22,073 - openselfsup - INFO - Set random seed to 0, deterministic: False
2021-01-06 13:12:22,334 - openselfsup - INFO - load model from: pth/byol_r50.pth
2021-01-06 13:12:30,498 - openselfsup - INFO - Start running, host: tangtao@lm01, work_dir: /data1/tangtao/OpenSelfSup/work_dirs/benchmarks/linear_classification/imagenet/r50_last/byol_r50.pth/train_20210106_131221
2021-01-06 13:12:30,498 - openselfsup - INFO - workflow: [('train', 1)], max: 100 epochs
2021-01-06 13:16:01,694 - openselfsup - INFO - head0_top1: 0.100
2021-01-06 13:16:01,696 - openselfsup - INFO - head0_top5: 0.530
2021-01-06 13:18:21,756 - openselfsup - INFO - Epoch [1][50/5005]	lr: 3.000e+01, eta: 16 days, 4:52:13, time: 2.797, data_time: 1.248, memory: 3223, loss: 6.9106, acc: 0.0625
2021-01-06 13:19:37,894 - openselfsup - INFO - Epoch [1][100/5005]	lr: 3.000e+01, eta: 12 days, 12:14:56, time: 1.523, data_time: 0.065, memory: 3223, loss: 6.9092, acc: 0.1328
2021-01-06 13:21:26,304 - openselfsup - INFO - Epoch [1][150/5005]	lr: 3.000e+01, eta: 12 days, 12:35:42, time: 2.168, data_time: 1.578, memory: 3223, loss: 6.9099, acc: 0.0625

@Trent-tangtao  Hi, have you solved the problem? If not, please refer to the following.
First, please try to use this model selfsup/byol/r50_bs2048_accumulate2_ep200_fp16.py (@xieenze), download link. This is the latest and performance is better.
Second, run byol with latest code and model, following the script:
bash benchmarks/dist_train_linear.sh configs/benchmarks/linear_classification/imagenet/r50_multihead.py   byol_r50_bs2048_accmulate2_ep200-e3b0c442.pth
Third, the result is as below, it looks good:

Please try this, hope to help you solve the problem.

@Trent-tangtao  Hi, have you solved the problem? If not, please refer to the following.
First, please try to use this model selfsup/byol/r50_bs2048_accumulate2_ep200_fp16.py (@xieenze), download link. This is the latest and performance is better.
Second, run byol with latest code and model, following the script:
bash benchmarks/dist_train_linear.sh configs/benchmarks/linear_classification/imagenet/r50_multihead.py   byol_r50_bs2048_accmulate2_ep200-e3b0c442.pth
Third, the result is as below, it looks good:

Please try this, hope to help you solve the problem.

Thanks again for your help! I have solved this problem with latest code. Maybe I accidentally changed the previous code, sorry for bothering you.

Thanks again for your help! I have solved this problem with latest code. Maybe I accidentally changed the previous code, sorry for bothering you.


emacs mode wraps comments incorrectly

 Issue by brson
Thursday Jan 22, 2015 at 20:42 GMT
For earlier discussion, see https://github.com/rust-lang/rust/issues/21524
This issue was labelled with: A-tools in the Rust repository

This is a regression that has existed for a while. If I enter a long line of comments:
    // The reason for the current stability level. If deprecated, the reason for deprecation.

and press Alt-Q it does this:
    // The reason for the current stability level. If deprecated, the
    reason for deprecation.

which is invalid syntax.

I cannot reproduce this with rust-mode 20150215.1646.

I also can't reproduce it with 20150215.1646.

Same, with commit 249e3cef5d3dacfec1c1947637d009dafa0fad8d (I don't know where these fancy version numbers come from :)

The version numbers come from MELPA.  It assigns a version based on the date of the commit that it pulled.  20150215.1646 I believe corresponds to commit 97f6445.
I went to look at what Marmalade does...and it looks like that relies on the author to upload the package, which hasn't been done for rust-mode in quite a while.  I'll put up another issue about that, since the README still talks about installing from Marmalade as well as MELPA.
As for this issue, if nobody can reproduce it, should it be closed?

Possibly related: While I get correct wrapping with fill-paragraph, I get the same result as @brson with fill-region.
If I add this to a brand new rust-mode buffer:
// The reason for the current stability level. If deprecated, the reason for deprecation.
// The reason for the current stability level. If deprecated, the reason for deprecation.
// The reason for the current stability level. If deprecated, the reason for deprecation.

and I run mark-whole-buffer and fill-region, I get:
// The reason for the current stability level. If deprecated, the reason for
deprecation.  // The reason for the current stability level. If deprecated, the
reason for deprecation.  // The reason for the current stability level. If
deprecated, the reason for deprecation.

but when I run the same two commands in c-mode, I get:
// The reason for the current stability level. If deprecated, the reason for
// deprecation.  The reason for the current stability level. If deprecated, the
// reason for deprecation.  The reason for the current stability level. If
// deprecated, the reason for deprecation.

(which is correct and identical to fill-paragraph in rust-mode)
Should this be in a new issue? (and can you all reproduce it?)

Yes, I can reproduce the issue as @utkarshkukreti explained it.  fill-paragraph (as long as the mark isn't active) but fill-region does not work (even if only one line is selected.)

I also have to create a region.
On Mar 6, 2015 4:18 PM, "Micah Chalmer"<EMAIL_ADDRESS>wrote:

Yes, I can reproduce the issue as @utkarshkukreti
https://github.com/utkarshkukreti explained it. fill-paragraph (as long
as the mark isn't active) but fill-region does not work (even if only one
line is selected.)
—
Reply to this email directly or view it on GitHub
https://github.com/rust-lang/rust-mode/issues/15#issuecomment-77660316.


WIP: cudaPackages.cudatoolkit: split outputs

Description of changes
Split outputs of cudaPackages.cudatoolkit so that the actual toolkit does not bring in dependencies of graphical/interactive apps (e.g. the debugging tools such as nsight) into the closure.
Note this is a work in progress

Here's the plan: I could create an intermediate derivation, say let cudatoolkit-src = ...; in, which would would only have the unpackPhase (same as one in current expression) and an installPhase that moves stuff to $out verbatim, without any patchelf. Then I'd rewrite cudatoolkit to take src = cudatoolkit-src, to extract the actual toolchain without nsight, and to apply autoPatchelf. Similarly, I can have a separate expression for nsight with src = cudatoolkit-src.
This way the outPath of cudatoolkit-src never really changes, and cudatoolkit does not depend on python2 or gtk3
@samuela, thoughts?

I'm not against this, but I imagine that it will eventually be made obsolete by the redist packages anyhow... So I guess it all comes down to how you'd prefer to spend your refactoring time :P

I'm not against this, but I imagine that it will eventually be made obsolete by the redist packages anyhow... So I guess it all comes down to how you'd prefer to spend your refactoring time :P

I just thought I'd pull this one off in no time, but in the end I can only afford occasional small contributions =
I'm closing this and will prioritize migrating pytorch/tf/jaxlib to redist instead


Fix gradle cache - DO NOT MERGE

Fixes #325
Possibly makes this support the Gradle cache with some minor caveats.

Doesn't support Windows
Possibly reduces compatibility with pre-Gradle 4.0 but should still work
It takes a long time to run (~45s with lots of dependencies)
Rather naive at this point

Note: It needs some cleanup, and I'm hoping to get some tests from others before this gets merged

@artur-shaik mind merging this? It's been working fine locally and no one else seems to mind. I can improve this as time goes on
Sent from my OnePlus ONEPLUS A3000 using FastHub

Ok, thanks.


[WIP] Add kafka consumer

Issue Ref: None
Description:
This PR adds an experiment kafka consumer which listens to messages published in a specific kafka topic. This is part of the trigger decoupling story that we would like to have independent trigger listeners which consume and forward messages to function. Right now this PR creates a kafka consumer deployment which will consume messages from the existing kafka/zookeeper setup and forward them to stdout. It's testable via the added integration test script and yaml manifest. There are still few things need to be done: 1) once the function handler is setup, kafka consumer should forward messages to it's endpoint. 2) if we would like to have different deployment model for consumer(s), then I will update this PR accordingly. Right now it's just a separate k8s deployment.
TODOs:

[x] Ready to review
[x] Automated Tests
[ ] Docs

@ngtuna was this merged in the triggers branch? shall we close this pr?

@arapulido Yes this PR can be closed as I've switched the work to trigger branch. When we all feel good with all tests in trigger branch, I will create a new PR to push changes to master.


Add ChatWebsocket example

as discussed in https://github.com/IronsDu/brynet/issues/58

@IronsDu could you check why did the CI build failed. Looks like it's related to the test on windows OS. I don't see how that could be related to my modifications.


chore: release core packages v0.2.2

Bump core packages to version 0.2.2 and generate changelogs.

CI / Test (macos-latest, 20) (pull_request) Failing after 29s

Network issues on Github actions, I think. Let's retry.

Seeing some random timeout errors on Github Actions when dependencies are loaded and errors from node-gyp. Let's retry this one again later today.


Invite Circle v3 to Slack

Invite the new Catalyst Circle v3 members to Slack:

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

<EMAIL_ADDRESS>(from the spreadsheet) bounced so I sent Rhys' invite to<EMAIL_ADDRESS>instead.

Email invites sent to all CCv3 members in a single email with the invite link using the<EMAIL_ADDRESS>account.


Add editorconfig file and standardise custom metric files

Adds the .editorconfig file as used in the Almanac for those using EditorConfig to help enforce file format styles.
Including spaces over tabs (let battle commence!)
To actually make use of this may require installing the editorconfig node module via npm (we suggest installing globally - npm install -g editorconfig ) and then installing the extension for your IDE (for example there is a Visual Studio Code extension).

LGTM


'MainWindow' object has no attribute 'checkBoardPB'

Mesact Version: 2.1.2 Build Date: 1/24/2024
File "../../usr/bin/mesact", line 120, in
ex = MainWindow()
File "../../usr/bin/mesact", line 75, in init
startup.setup(self)
File "../../usr/lib/python3/dist-packages/libmesact/startup.py", line 124, in setup
parent.checkBoardPB.setEnabled(False)
AttributeError
'MainWindow' object has no attribute 'checkBoardPB'
First download the deb and reinstall
to be sure you have the latest build date
Then please file an issue at
https://github.com/jethornton/mesact/issues

Mesact Version: 2.1.1 Build Date: 1/18/2024
File "../../usr/bin/mesact", line 120, in
ex = MainWindow()
File "../../usr/bin/mesact", line 75, in init
startup.setup(self)
File "../../usr/lib/python3/dist-packages/libmesact/startup.py", line 111, in setup
parent.checkBoardPB.setEnabled(False)
AttributeError
'MainWindow' object has no attribute 'checkBoardPB'
First download the deb and reinstall
to be sure you have the latest build date
Then please file an issue at
https://github.com/jethornton/mesact/issues

2.1.0 Same thing. 2.0.8 Works

satiowadahc, download and install mesaflassh here:  https://github.com/LinuxCNC/mesaflash and JT will fix the bug this afternoon. He is unavailable at the moment.


Add explicit dependency on libfuzzer-sys

This fixes a build issue with the most recent version of libfuzzer-sys where the prior version of the fuzz_target! macro didn't need the crate as an explicit dependency but the latest version does.

The fuzz build also currently prints out:
warning: for loop over a `Result`. This is more readably written as an `if let` statement
  --> fuzz_targets/moves.rs:65:18
   |
65 |         for i in u.int_in_range(0..=2) {
   |                  ^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(for_loops_over_fallibles)]` on by default
help: to check pattern in a loop use `while let`
   |
65 |         while let Ok(i) = u.int_in_range(0..=2) {
   |         ~~~~~~~~~~~~~ ~~~
help: consider using `if let` to clear intent
   |
65 |         if let Ok(i) = u.int_in_range(0..=2) {
   |         ~~~~~~~~~~ ~~~

which is just a warning so not fatal, but I wasn't sure what the intended code here was so I figured I'd ask.

What libfuzzer-sys change caused this to be necessary? I'd expect this to be a bug in that crate, not something that should be worked around here. I think libfuzzer-sys 0.4.3 should have made this unnecessary, so I'd like to know what broke after that.
In #69 I removed this dependency because it requires keeping dependencies in sync across multiple Cargo.toml files. So I'd like to revert this PR but clearly we need something.

I'm not sure what the change was, I mostly just wanted to fix the fuzz build, I didn't investigate much futher.