help getting started

First..  thanks so much for sharing your Dockerfile for building Jena with geosparql support.
I am able to both pull and run and also rebuild the image with podman on Fedora.
When running I get the following:
(base) ➜  fuseki-geosparql git:(main) podman run -i --privileged  -v /home/fils/Containers/dvols/jena:/data -p 3030:3030 --rm a9e5420065a9
[opentelemetry.auto.trace 2022-01-21 16:50:08:092 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.2.0
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
[fuseki 2022-01-21 16:50:09:375 +0000] [main] INFO org.apache.jena.fuseki.geosparql.Main {} - Arguments Received: [--loopback, false, -t, /fuseki/databases/ds, -t2, --validate, --default_geometry]
[fuseki 2022-01-21 16:50:09:411 +0000] [main] INFO org.apache.jena.fuseki.geosparql.DatasetOperations {} - Server Configuration: port=3030, datsetName=ds, loopbackOnly=false, updateAllowed=false, inference=false, applyDefaultGeometry=true, validateGeometryLiteral=true, convertGeoPredicates=false, removeGeoPredicates=false, queryRewrite=true, tdbFile=/fuseki/databases/ds, fileGraphFormats=[], fileGraphDelimiters=[], indexEnabled=true, indexSizes=[-1, -1, -1], indexExpiries=[5000, 5000, 5000], spatialIndexFile=null, tdb2=true, help=false
[fuseki 2022-01-21 16:50:09:465 +0000] [main] INFO org.apache.jena.fuseki.geosparql.DatasetOperations {} - TDB Dataset: /fuseki/databases/ds, TDB2: true
[fuseki 2022-01-21 16:50:10:031 +0000] [main] INFO org.apache.jena.geosparql.configuration.GeoSPARQLOperations {} - Applying hasDefaultGeometry - Started
[fuseki 2022-01-21 16:50:10:079 +0000] [main] INFO org.apache.jena.geosparql.configuration.GeoSPARQLOperations {} - Applying hasDefaultGeometry - Completed
[fuseki 2022-01-21 16:50:10:185 +0000] [main] WARN org.apache.sis.system {} - The “SIS_DATA” environment variable is not set.
[fuseki 2022-01-21 16:50:10:351 +0000] [main] WARN org.apache.jena.fuseki.geosparql.DatasetOperations {} - Datset empty. Spatial Index not constructed. Server will require restarting after adding data and any updates to build Spatial Index.
[fuseki 2022-01-21 16:50:10:761 +0000] [Thread-4] INFO org.apache.jena.fuseki.geosparql.GeosparqlServer {} - GeoSPARQL Server: Running - Port: 3030, Dataset: /ds, Loopback Only: false,  Allow Update: false
[fuseki 2022-01-21 16:50:10:841 +0000] [Thread-4] INFO org.apache.jena.fuseki.Server {} - Start Fuseki (http=3030)

I see the updates are set to false which I can confirm with
mc cat nas/issue31/summoned/invemarexpert//test.jsonld | jsonld format -q | curl -X PUT --header "Content-Type:application/n-quads" -d @- http://localhost:3030/ds/data
HTTP method not allowed: PUT : Read-only

So it seems the server is running and I can get the "read-only" warning when trying to load some data, which I expect.
hitting:  http://localhost:3030/
results in Error 404: Not Found
I do have config files and approaches that work for me with core Jena 4  (no geosparql) but I am not exactly sure where to mount and leverage my config files.
Do you have some documentation on firing up the container with a mounted volume to allow setting up a read-write database that I could start to load in some geosparql enhanced RDF into?
Again.  thanks so much for this.   I was not able to get even close to this far with building in geosparql support!
Any help you can provide would be appreciated!

Can't comment on the error, my colleague @ludovicm67 might though. But just as a FYI, Andy is working on better integration into Fuseki. You might want to play with that, see https://issues.apache.org/jira/browse/JENA-2230

Hi @fils! Thanks for showing interest in our Fuseki Docker image with GeoSPARQL support.
I was waiting that the 4.4.0 version of Apache Jena was released, which is now the case.
For context, we are using this Docker image for some read-only cases, so this was perfectly fitting our needs on that part.
But we really needed to have at some point a way to do some configuration using configuration files, which was not possible before this release.
I will spend some time in the next days on working on this Docker image to have it using the latest version of Apache Jena and see how to have something great in terms of configuration, and see what will be possible to do or not.
Everything will be documented, and I will give some updates here once I'm done.

@ludovicm67 great news!
I'll be looking forward to seeing it soon and giving it a try!  Really appreciate you sharing this development work!

Hi @fils! I worked on upgrading the Fuseki version and add support for the WebUI and configuration options.
It's now ready, and you can get some documentation in the README.md file.
Some more additional information about Fuseki with GeoSPARQL: https://jena.apache.org/documentation/geosparql/.
Can you give it a try?
Please ping me if something is unclear so that I can update the README file consequently.
If everything is fine, we can close this issue.

@ludovicm67  exciting!  I will try and give it a try here this evening and report back!

@ludovicm67 sorry for the delay in getting back to you.   Here is where I am at.
I can successfully build and run the repo.   However, I am having two issues.   First, using
(base) ➜  fuseki-geosparql git:(main) ✗ podman run -i --privileged  -e ADMIN_PASSWORD -v /home/fils/Containers/dvols/jena:/fuseki/databases -p 3030:3030 --rm ab1169ffb4f6

[otel.javaagent 2022-02-11 13:58:30:914 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.10.1
[fuseki 2022-02-11 13:58:32:847 +0000] [main] WARN org.apache.sis.system {} - The “SIS_DATA” environment variable is not set.
[fuseki 2022-02-11 13:58:33:415 +0000] [main] INFO org.apache.jena.fuseki.Server {} - Apache Jena Fuseki 4.4.0
[fuseki 2022-02-11 13:58:33:583 +0000] [main] INFO org.apache.jena.fuseki.Config {} - FUSEKI_HOME=/opt/fuseki
[fuseki 2022-02-11 13:58:33:583 +0000] [main] INFO org.apache.jena.fuseki.Config {} - FUSEKI_BASE=/fuseki
[fuseki 2022-02-11 13:58:33:619 +0000] [main] INFO org.apache.jena.fuseki.Config {} - Shiro file: file:///fuseki/shiro.ini
[fuseki 2022-02-11 13:58:34:427 +0000] [main] INFO org.apache.jena.geosparql.configuration.GeoSPARQLOperations {} - Applying GeoSPARQL Schema - Started
[fuseki 2022-02-11 13:58:34:438 +0000] [main] INFO org.apache.jena.geosparql.configuration.GeoSPARQLOperations {} - GeoSPARQL schema not applied to empty graph: default
[fuseki 2022-02-11 13:58:34:454 +0000] [main] INFO org.apache.jena.geosparql.configuration.GeoSPARQLOperations {} - Applying GeoSPARQL Schema - Completed
[fuseki 2022-02-11 13:58:34:473 +0000] [main] WARN org.apache.jena.geosparql.assembler.GeoAssembler {} - Dataset empty. Spatial Index not constructed. Server will require restarting after adding data and any updates to build Spatial Index.
[fuseki 2022-02-11 13:58:34:515 +0000] [main] WARN org.apache.jena.fuseki.server.EndpointSet {} - Redefining endpoint for query
[fuseki 2022-02-11 13:58:34:657 +0000] [main] INFO org.apache.jena.fuseki.Server {} - Configuration file: /fuseki/config.ttl
[fuseki 2022-02-11 13:58:34:658 +0000] [main] INFO org.apache.jena.fuseki.Server {} - Path = /ds
[fuseki 2022-02-11 13:58:34:658 +0000] [main] INFO org.apache.jena.fuseki.Server {} - System
[fuseki 2022-02-11 13:58:34:660 +0000] [main] INFO org.apache.jena.fuseki.Server {} -   Memory: 7.8 GiB
[fuseki 2022-02-11 13:58:34:660 +0000] [main] INFO org.apache.jena.fuseki.Server {} -   Java:   16.0.2
[fuseki 2022-02-11 13:58:34:660 +0000] [main] INFO org.apache.jena.fuseki.Server {} -   OS:     Linux 5.15.11-200.fc35.x86_64 amd64
[fuseki 2022-02-11 13:58:34:661 +0000] [main] INFO org.apache.jena.fuseki.Server {} -   PID:    1
[fuseki 2022-02-11 13:58:34:799 +0000] [main] INFO org.apache.jena.fuseki.Server {} - Started 2022/02/11 13:58:34 GMT on port 3030

I am using podman, but hopefully that is not introducing any issues.
following
ARG FUSEKI_BASE="/fuseki"
RUN mkdir -p "${FUSEKI_BASE}/databases"

I mount /fuseki/databases but I never see any DB written to it.
When loading I used:
(base) ➜  jena  curl -X PUT --header "Content-Type:application/n-triples" -d @edmotest.rdf http://localhost:3030/ds/data
{
  "count" : 25 ,
  "tripleCount" : 25 ,
  "quadCount" : 0
}

the server shows

[fuseki 2022-02-11 14:07:15:168 +0000] [qtp1320631496-42] INFO org.apache.jena.fuseki.Fuseki {trace_id=32ac39496717f14abd1e3fde252b5e78, span_id=91b323ea785a053a, trace_flags=01} - [14] PUT http://localhost:3030/ds/data
[fuseki 2022-02-11 14:07:15:178 +0000] [qtp1320631496-42] INFO org.apache.jena.fuseki.Fuseki {trace_id=32ac39496717f14abd1e3fde252b5e78, span_id=91b323ea785a053a, trace_flags=01} - [14] Body: Content-Length=3717, Content-Type=application/n-triples, Charset=null => N-Triples : Count=25 Triples=25 Quads=0
[fuseki 2022-02-11 14:07:15:181 +0000] [qtp1320631496-42] INFO org.apache.jena.fuseki.Fuseki {trace_id=32ac39496717f14abd1e3fde252b5e78, span_id=91b323ea785a053a, trace_flags=01} - [14] 200 OK (12 ms)

I modified the config file
...
<#service> rdf:type fuseki:Service;
  fuseki:name "ds";

  fuseki:serviceQuery               "query" ;    # SPARQL query service
  fuseki:serviceUpdate              "update" ;   # SPARQL update service
  fuseki:serviceUpload              "upload" ;   # Non-SPARQL upload service
  fuseki:serviceReadWriteGraphStore "data" ;     # SPARQL Graph store protocol (read and write)

  fuseki:endpoint [ fuseki:operation fuseki:query ; ];
  fuseki:endpoint [ fuseki:operation fuseki:query ; fuseki:name "sparql" ];
  fuseki:endpoint [ fuseki:operation fuseki:query ; fuseki:name "queryold" ];
  fuseki:endpoint [ fuseki:operation fuseki:gsp-r ; ];
  fuseki:endpoint [ fuseki:operation fuseki:gsp-r ; fuseki:name "get" ];

  fuseki:dataset <#ds> .
...

perhaps I did that wrong?
This is where I am at so far.  If you see anything I am doing wrong or have any guidance I would appreciate it.
Thanks
Doug

Update.   It looks like I should be using POST, not PUT.
Also, I still don't seem able to write to ds yet.  However if I log in and make a new database it does allow me to load and persist the triples.

Update 2
podman run -i --privileged  -e ADMIN_PASSWORD -v rdfdata:/fuseki -p 3030:3030 --rm ab1169ffb4f6

is working better.  I can create a docker volume mount it and then copy in the config files to the /fuseki mount point.  This allows me to persist the data through booting.   I think I just need to modify the config files for my databases there and I should have a working approach.
This might be nice too since I might be able to mount it read only to a public version and read write to a controlled version.  We will see.
But I am getting closer.  :)

@fils thanks for your feedback!
I updated the default configuration in the Docker image to have the default dataset ds stored in /fuseki/databases/ds. Previously it was in /fuseki/ds.
So you can mount a volume at /fuseki/databases to persist all your datasets.
You can edit the configuration file which is mounted at /fuseki/config.ttl.
What's great with Fuseki, is that you can easily configure new endpoints using the configuration file.
You can decide to have read-only endpoints that you can share with everyone, and read-write ones that requires a password for example.
To configure some restricted access, you can edit the shiro.ini file.
Also, a thing to know is that the spatial index will only be built when Fuseki is starting.
If you do many changes in the dataset, a good thing can be to restart the instance, so that the spacial index is up-to-date and so you can expect better performances.

@ludovicm67 Thanks for the update..  I'll pull the changes and work from that.
I made good progress on testing and will report back with the new version.
I was able to make a test database, load some RDF with spatial information and do a successful geosparql call!   So very happy with that.

@fils can I close this issue or do you have any other related question?

@ludovicm67 go ahead and close the issue.   I have a working build and I can do the mounts to persist between running the containers.   I might have a few questions about the direct layout of the directory and the config files but if I do I can open a new issue.
Same re: the access control approaches you mention in your previous post.  I have no expectation of issues with that, just never configured that.   So I may get lost.  ;)
For the scope of this issue though I am fine.
Thanks again for all your work on this and for sharing it!   Very much appreciated.

@fils thanks for all of your feedback!
Yes, feel free to open any new issue if needed, this can also help others to use the Docker image :)


Realtime: implement ARTConnection & ARTConnectionDetails

[ ] Connection
[ ] ConnectionStateChange
[ ] ConnectionDetails

@tcard you mentioned you have worked on implementing the correct types, will it address any of the above?

Not really, no. I basically took a look at Message and related types, but didn't get anything done beyond a few notes, I had been mainly reading stuff (Ably code, Ably docs and iOS docs).


SQL Transaction is completed but it's no longer is usable

Hi,
Today I am getting one error while executing scripts using DBUp Migration tool. It seems all scripts are executed and also found in the journal table.
When I re-executed, it's successfully done  ( because all scripts skipped, found records of these into journal table)
Can you please help me to understand why this error is coming and causing this error.

Sorry no. But perhaps you can ask the folks over at DbUp? I am guessing it has nothing to do with this task due to the stack trace.

Can you try and use TransactionPerScript and see if that works?

Closing due to lack of activity...


Change Collection.creators to Collection.contributors

Change the Collection.creators field to Collections.contributors, including updating all existing Collection data.

MATCH (c:Collection) WHERE c.creators IS NOT NULL SET c.contributors = c.creators REMOVE c.creators


add profiles

F1 -  F6 can be used to toggle between saved profiles. F1 is the default profile loaded when program is first started.

resolves #2


Fix peso

you could do something like this: https://github.com/yearn/yearn-exporter/blob/9f3117b14ac9b4df37578ec7f36b5df0ab8e5a0a/yearn/init.py#L20-L24

The dependency is already going to be included as part of #420, and after that point it would make sense to use the convenience method provided by ypricemagic.

merging after discussion with @mariuspod


Unable to use composite build

Expected Behavior
If something is not supported by composite builds, it should minimally provide an error message which makes sense to the user. The Micronaut projects consist of a lot of different repositories and I'm trying to setup a composite in order to ease our development process, for example to make sure that a change to core doesn't impact downstream consumers.
Current Behavior
The one I'm getting is impossible to interpret:
* What went wrong:
Could not determine the dependencies of task ':micronaut-data:buildSrc:test'.
> Could not determine the dependencies of null.
   > Could not resolve all dependencies for configuration ':micronaut-data:buildSrc:runtimeClasspath'.
      > Could not resolve all dependencies for configuration 'incomingCatalogForMnData0'.
         > Expected vintage state of build 'micronaut-data' to be in state SettingsLoaded or later but is in state SettingsLoaded and transitioning to Configured.

Context
I'm trying to setup an "uber build" for Micronaut, which is a composite of multiple repositories. Depending on which projects I'm including, the build may or may not pass. The problem seems to be related to cycles in composites. For example, micronaut-test-resources depends on micronaut-serde which is also included in the composite. If I remove either of these, the build passes.
Steps to Reproduce

Clone https://github.com/melix/micronaut-uber-build
Checkout branch composite-error-1 or composite-error-2 (both are different configurations which both exercise the same bug)
run ./gradlew -I disable-broken.gradle test (the init script is meant to disable POM publishing tasks which fail with an NPE, but it's a different bug)

Your Environment
Gradle 7.6
Build scan URL: https://ge.micronaut.io/s/wnn5vah24lcda

Thank you for providing a valid reproducer.
The issue is in the backlog of the relevant team but the existence of a workaround makes it non-critical so it might take a while before a fix is made.

Thanks Louis, but there is actually no workaround :) The build simply doesn't work right now, since depending on the combinations of modules I'm facing different issues.

My bad, I thought you said that removing some modules fixed it. Removing the label. And updating my answer.

I hit this error as well on gradle 8.2.1.
if I add this 👇  to build-logic/included-build/settings.gradle.kts it triggers the Expected vintage state of build... error in the composite build that is pulling that in as an included build.
plugins {
    kotlin("jvm") version(embeddedKotlinVersion) apply(false)
}

If I remove it, the error goes away.
I wasn't using buildSrc for anything else, but adding this seem work workaround this issue:
build-logic/included-build/buildSrc/build.gradle.kts
dependencies {
    implementation(embeddedKotlin("gradle-plugin"))
}


Add transaction/retry support for yii\db\Command

Q
A

Is bugfix?
no

New feature?
yes

Breaks BC?
no

Tests pass?
yes

Fixed issues
see #15276

It's required to perform upsert statements safely and securely.
This PR has to be merged before #15276.

I haven't found the retry part in this PR (or I didn't get it). But I wanted to link to this retry lib for PHP, which is really great - just in case it might be useful here.

@schmunk42 Well, it works the same, you may want to start here for an usage example.

Merged. Thank you!


ARROW-8020: [Java] Implement vector validate functionality

Related to ARROW-8020.
In C++ side, we already have array validate functionality but no similar functionality in Java side.
This issue is about to implement this functionality.
see ML discussion
https://lists.apache.org/thread.html/r5e463275b24dd171cb0347adb8dc10a07d5de97cfa6e023b7806db55%40<dev.arrow.apache.org>

cc @emkornfield

+1 thank you @tianchen92


Add badges

We add a couple of badges for visibility.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>7
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 91.111%

Totals

Change from base Build<PHONE_NUMBER>4:
0.0%

Covered Lines:
41

Relevant Lines:
45

💛  - Coveralls


Getting "IllegalLocationConstraintException" on 's3.create_bucket'

Getting an exception when trying to create a bucket in a mock_s3'd test in pytest. These exact tests were passing fine 3 days before this ticket was created. Even installing the same versions of packages of when the tests were passing doesn't help: the same error occurs.
In addition, if I don't have a default region set, I get "BucketAlreadyExists", but with a default region set, I get "IllegalLocationConstraintException".
This has been verified in a fresh Ubuntu Docker image.
 I dropped back to pytest 3.7.0 and the exception went away; see https://github.com/pytest-dev/pytest/issues/3774.

It may also be a regression with botocore as well

Seeing similar here on django testing without pytest.  I suspect this is a botocore related issue.

It seems to be related to botocore yes.

I am also experiencing this issue. Furthermore, it seems that this causes the tests to actually create the test bucket in S3.

Was seeing this same problem until I dropped down boto3 versions.
"boto3>=1.6.16,<1.8",
"botocore>=1.9.16,<1.11",

Also covered in this PR: https://github.com/spulec/moto/pull/1801

Going to close as a dupe of #1801. Let me know if that isn't the case.


关于markdown基础语法的问题

Describe the solution you'd like
首页要感觉开源社区无私的奉献，我无法为该项目做出贡献，深感惭愧，但至从0.18.2开始，基本的markdown语法支持就出现了问题，可以看想着的已经关闭的问题 #2983、#3242。
我因为没有参与项目，也不了解该问题是不是很难实现，但我很想知道为什么0.18.2之后会出现问题。因为我每天都需要该功能。所以我只能退回到0.18.1版本。
en:
The homepage wants to feel the selfless dedication of the open source community. I am deeply ashamed that I cannot contribute to the project. However, starting from 0.18.2, there are problems with basic markdown syntax support. You can look at the closed issues #2983, #3242.
Because I am not involved in the project, I don’t know whether this problem is difficult to implement, but I would like to know why the problem occurs after 0.18.2. Because I need this feature every day. So I can only go back to version 0.18.1.
Type of feature
User Experience (UX)
Additional context
如果markdown功能 能回到0.18.1的功能，我觉得已经很好了。不用再改进什么了，因为标签的重复在日常的零碎记录中很有用。这本身就是一个文本记录，也不能要求能做表格或其它更复杂的markdown。
一次回车是重复标签。再次回车是清除重复的标签。这对于记录零散事项很有用。
en:
If the markdown function can return to the function of 0.18.1, I think it will be very good. There is no need to improve anything, because the repetition of tags is very useful in daily fragmentary recording. This is a text record in itself, and it cannot be required to be a table or other more complex markdown.
A carriage return is a duplicate tag. Press Enter again to clear duplicate tags. This is useful for keeping track of loose ends.

今天我刚体验了0.22.0版本，发现问题更大。带列表标签的行回车后会自动增加了一个空行。经过0.19、0.20、0.21、0.22的使用之后。我只能先退回0.18.1了。其它的展示上的美观其实对于我来说没那么重要。真正的是使用上的便利才更重要。
en ：
I just tried version 0.22.0 today and found that the problem is even bigger. A blank line is automatically added to a line with a list label after a carriage return. After using 0.19, 0.20, 0.21, 0.22. I can only return to 0.18.1 first. The aesthetics of other displays are actually not that important to me. What is really more important is the convenience of use.


Flexible pod VM instance sizes

Today the pod VM instance size is fixed. Creating this issue to discuss on how how we can make it variable depending on the Pod cpu and memory requests and limits?
One possible approach is for the cloud admin to configure a list of pod vm instance types, including a default instance size.
CAA or mutation webhook takes the closest match based on the cpu and memory resource request/limit.
When no cpu and memory requests/limits are present in the pod spec, the default instance type is used.
The advantage of this approach is that it avoids keeping track of different cloud instance types. However it comes with limited flexibility.
Any other approaches ?

cc @yoheiueda @stevenhorsman @huoqifeng @knrt10 @surajssd @mkulke

One possible approach is for the cloud admin to configure a list of pod vm instance types, including a default instance size.
CAA or mutation webhook takes the closest match based on the cpu and memory resource request/limit.

I think for the first iteration this is a reasonable approach. Where would such a admin-curated list be stored? I suppose it has to be a tuple (instancetype, vcpu, mem) to match a workload to an instance. Adding a list to the cli invocation would require embedding this information in the argument somehow:
--instance-types Standard_DC2as_v5:2:8;Standard_DC4as_v5:4:16
alternatively vcpus and memory could be retrieved from an api call and cached, not sure whether that works for all providers, though.

I like the tuple idea!
For now this is good, but such configs on cmd line, begs the question if we need a CRD for CAA?
Implementation wise, we can try to match the closest upper limit (or say ceiling) of a machine we have from the list. Like if a user requests 1cpu and 1GB memory and the only closest machine from the list we have is Standard_DC2as_v5:2:8 we can use that.
Above approaches are good for lift and shift use cases, where a user brings in existing Deployment objects and things work OOTB without any config change. Although I wonder how many of the workloads will really be lift and shift! Because users would have to change things like node selectors, runtimeclass and such to influence scheduling of pods on encrypted hardware machine types.
For advanced users, who wants to provide a machine type themselves, we should allow them to provide annotations for machine types on the lines of io.cc.caa.machinetype=Standard_DC4as_v5.

I like the tuple idea!
For now this is good, but such configs on cmd line, begs the question if we need a CRD for CAA?

Increasingly the need for a CRD is felt now.  Consider the case of cloud resource pruning in case of any errors. CAA is stateless today and there are issues in the error path - node maintenance, CAA restart etc.
May be we can piggy back on @jensfr work - https://github.com/confidential-containers/cloud-api-adaptor/issues/328 ?

Above approaches are good for lift and shift use cases, where a user brings in existing Deployment objects and things work OOTB without any config change. Although I wonder how many of the workloads will really be lift and shift! Because users would have to change things like node selectors, runtimeclass and such to influence scheduling of pods on encrypted hardware machine types.

Changing the spec can be also done automatically using policy engines like opa or kyverno. So solutions exist which can be customised as needed by the cluster admin.

For advanced users, who wants to provide a machine type themselves, we should allow them to provide annotations for machine types on the lines of io.cc.caa.machinetype=Standard_DC4as_v5.

Makes sense.

@jtumber-ibm I have some poc code that I'm currently working on. Should be able to share an early draft in next few days


Disable automatic writeback of new Microsoft 365 groups - Does this work for sovereign clouds?

For the section labeled "Disable automatic writeback of new Microsoft 365 groups", my testing does not show this capability working in the Entra ID US Government cloud.
Can you verify if the NewUnifiedGroupWritebackDefault directory setting is available in the Government cloud?  If not, can timeline on when to expect it?
Also, regardless, a note should be added to prevent further confusion.
Thanks!

@ShahMoneyXL
It would be great if you could add a link to the documentation you are following for these steps? This would help us redirect the issue to the appropriate team. Thanks!

@ShahMoneyXL
We are going to close this thread, if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.
#Please-close


Update monaco-editor to version 0.34.0

@suren-atoyan Is this PR ok by you? (GitHub does not let me add you as a reviewer for some reason)

hey @Urook, yes, it looks good - I'll merge it now, but it will take some time to publish it

@Urook also, be aware that you can always choose the exact version you need by using config:

import loader from '@monaco-editor/loader';

loader.config({ paths: { vs<EMAIL_ADDRESS>} });

Thank you :)


Debugger fails to run after updating to MacOS Big Sur 11.5 on M1 MacBook Air

Issue Type: Bug
launch.json:
{
      "name": "Python: FastAPI",
      "type": "python",
      "request": "launch",
      "module": "uvicorn",
      "args": [
          "app.main:app",
          "--reload"
      ],
      "jinja": true
  }

Error:
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 311, in run_module
    start_debugging(argv_0)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 257, in start_debugging
    debugpy.connect(options.address, access_token=options.adapter_access_token)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/common/compat.py", line 208, in kwonly_f
    return f(*args, **kwargs)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/__init__.py", line 135, in connect
    return api.connect(address, access_token=access_token)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 143, in debug
    log.reraise_exception("{0}() failed:", func.__name__, level="info")
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 141, in debug
    return func(address, settrace_kwargs, **kwargs)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 276, in connect
    _settrace(host=host, port=port, client_access_token=access_token, **settrace_kwargs)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 47, in _settrace
    return pydevd.settrace(*args, **kwargs)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2696, in settrace
    _locked_settrace(
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2819, in _locked_settrace
    py_db.enable_tracing(py_db.trace_dispatch, apply_to_all_threads=True)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 1039, in enable_tracing
    pydevd_tracing.SetTrace(thread_trace_func)
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd_tracing.py", line 83, in SetTrace
    if set_trace_to_threads(tracing_func, thread_idents=[thread.get_ident()], create_dummy_thread=False) == 0:
  File "/Users/voxkingston/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd_tracing.py", line 349, in set_trace_to_threads
    result = lib.AttachDebuggerTracing(
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 378, in __getattr__
    func = self.__getitem__(name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 383, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, AttachDebuggerTracing): symbol not found

VS Code version: Code 1.58.2 (c3f126316369cd610563c75b1b1725e0679adfb3, 2021-07-14T22:10:12.490Z)
OS version: Darwin arm64 20.6.0
Restricted Mode: No

System Info

Item
Value

CPUs
Apple M1 (8 x 24)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
2, 1, 1

Memory (System)
8.00GB (0.13GB free)

Process Argv
--crash-reporter-id 4f75daaf-084f-4091-a930-8e8b022d84f1

Screen Reader
no

VM
0%

Extensions (19)

Extension
Author (truncated)
Version

vscode-sqlite
ale
0.13.0

vscode-database
baj
2.2.3

vscode-django
bat
1.6.0

gitlens
eam
11.6.0

vscode-dbml
mat
0.3.5

git-graph
mhu
1.30.0

dotenv
mik
1.0.1

vscode-docker
ms-
1.15.0

vscode-kubernetes-tools
ms-
1.3.3

python
ms-
2021.7.1053846006

vscode-pylance
ms-
2021.7.5

jupyter
ms-
2021.8.1054968649

remote-containers
ms-
0.187.1

vsliveshare
ms-
1.0.4583

vscode-commons
red
0.0.6

vscode-yaml
red
0.21.1

LiveServer
rit
5.6.1

pdf
tom
1.1.0

markdown-all-in-one
yzh
3.4.0

(1 theme extensions excluded)

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
pythonvspyt602:30300191
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt639:30300192
pythontb:30283811
pythonvspyt551cf:30311713
vspre833:30321513
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscus158:30321503
pythonvsuse255:30340121
vscod805cf:30301675
pythonvspyt200:30340761
vscextlang:30333561
binariesv615:30325510
pythonvssor306cf:30341832
bridge0708:30335490

I have the same issue, but I'm still on 11.4. Debugging works with conda environments, but not with venv and „m1 native” Python. python -m mymodule works (with venv automatically activated) in the integrated terminal. But with the config:
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "my module",
            "type": "python",
            "request": "launch",
            "module": "mymodule",
        }
    ]
}

Trying to debug via VSCode fails, but the venv path is correct:
cd /Users/fynnleitow/git/my-module ; /usr/bin/env /Users/fynnleitow/git/my-module/venv/bin/python /Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/launcher 57315 -- -m mymodule --help
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 311, in run_module
    start_debugging(argv_0)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 257, in start_debugging
    debugpy.connect(options.address, access_token=options.adapter_access_token)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/common/compat.py", line 208, in kwonly_f
    return f(*args, **kwargs)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/__init__.py", line 135, in connect
    return api.connect(address, access_token=access_token)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 143, in debug
    log.reraise_exception("{0}() failed:", func.__name__, level="info")
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 141, in debug
    return func(address, settrace_kwargs, **kwargs)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 276, in connect
    _settrace(host=host, port=port, client_access_token=access_token, **settrace_kwargs)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 47, in _settrace
    return pydevd.settrace(*args, **kwargs)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2696, in settrace
    _locked_settrace(
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2819, in _locked_settrace
    py_db.enable_tracing(py_db.trace_dispatch, apply_to_all_threads=True)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 1039, in enable_tracing
    pydevd_tracing.SetTrace(thread_trace_func)
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd_tracing.py", line 83, in SetTrace
    if set_trace_to_threads(tracing_func, thread_idents=[thread.get_ident()], create_dummy_thread=False) == 0:
  File "/Users/fynnleitow/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd_tracing.py", line 349, in set_trace_to_threads
    result = lib.AttachDebuggerTracing(
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 378, in __getattr__
    func = self.__getitem__(name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py", line 383, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, AttachDebuggerTracing): symbol not found

Similar situation is with me as well:
I guess the same problem is why test can't be debugged as well, although it doesn't show the same error (I've raised issue for that: #129221).
It might be something with the Python extension, which has been updated a couple of days ago. They been working on improving a debugger (changelog)
Error log:
Traceback (most recent call last):
  File<EMAIL_ADDRESS>line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File<EMAIL_ADDRESS>line 87, in _run_code
    exec(code, run_globals)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 267, in run_file
    start_debugging(target)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 257, in start_debugging
    debugpy.connect(options.address, access_token=options.adapter_access_token)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/common/compat.py", line 208, in kwonly_f
    return f(*args, **kwargs)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/__init__.py", line 135, in connect
    return api.connect(address, access_token=access_token)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 143, in debug
    log.reraise_exception("{0}() failed:", func.__name__, level="info")
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 141, in debug
    return func(address, settrace_kwargs, **kwargs)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 276, in connect
    _settrace(host=host, port=port, client_access_token=access_token, **settrace_kwargs)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 47, in _settrace
    return pydevd.settrace(*args, **kwargs)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2696, in settrace
    _locked_settrace(
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2819, in _locked_settrace
    py_db.enable_tracing(py_db.trace_dispatch, apply_to_all_threads=True)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 1039, in enable_tracing
    pydevd_tracing.SetTrace(thread_trace_func)
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd_tracing.py", line 83, in SetTrace
    if set_trace_to_threads(tracing_func, thread_idents=[thread.get_ident()], create_dummy_thread=False) == 0:
  File "/Users/solvebio/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd_tracing.py", line 349, in set_trace_to_threads
    result = lib.AttachDebuggerTracing(
  File<EMAIL_ADDRESS>line 387, in __getattr__
    func = self.__getitem__(name)
  File<EMAIL_ADDRESS>line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, AttachDebuggerTracing): symbol not found

I just started to have the same issue (11.3) after restarting the vscode 😅
{     "version": "0.2.0",     "configurations": [         {             "name": "Python: FastAPI",             "type": "python",             "request": "launch",             "module": "uvicorn",             "args": [                 "main:app",                 "--reload"             ],             "jinja": true         }     ] }
I can run the app using uvicorn successfully but when I want to debug it I get the following;
Traceback (most recent call last):
File<EMAIL_ADDRESS>line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File<EMAIL_ADDRESS>line 87, in _run_code
exec(code, run_globals)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/main.py", line 45, in
cli.main()
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
run()
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 311, in run_module
start_debugging(argv_0)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 257, in start_debugging
debugpy.connect(options.address, access_token=options.adapter_access_token)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/common/compat.py", line 208, in kwonly_f
return f(*args, **kwargs)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/init.py", line 135, in connect
return api.connect(address, access_token=access_token)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 143, in debug
log.reraise_exception("{0}() failed:", func.name, level="info")
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 141, in debug
return func(address, settrace_kwargs, **kwargs)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 276, in connect
_settrace(host=host, port=port, client_access_token=access_token, **settrace_kwargs)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/../debugpy/server/api.py", line 47, in _settrace
return pydevd.settrace(*args, **kwargs)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2696, in settrace
_locked_settrace(
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2819, in _locked_settrace
py_db.enable_tracing(py_db.trace_dispatch, apply_to_all_threads=True)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 1039, in enable_tracing
pydevd_tracing.SetTrace(thread_trace_func)
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd_tracing.py", line 83, in SetTrace
if set_trace_to_threads(tracing_func, thread_idents=[thread.get_ident()], create_dummy_thread=False) == 0:
File "/Users/onur/.vscode/extensions/ms-python.python-2021.7.1053846006/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd_tracing.py", line 349, in set_trace_to_threads
result = lib.AttachDebuggerTracing(
File<EMAIL_ADDRESS>line 387, in getattr
func = self.getitem(name)
File<EMAIL_ADDRESS>line 392, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, AttachDebuggerTracing): symbol not found

I have the same issue with Big Sur 11.4 and VS Code. Yesterday everything was fine. Today I get this error when I try to start debugging.

Started getting same issue today in VS Code, Big Sur 11.2.3

Same issue, Big Sur 11.4 and VS Code. Worked yesterday, doesn't work today.

Glad to know I'm not the only one. Same issue. Big Sur 11.4, M1 chip, VS Code.

/extPython

Hi, get same problem today. Seems it some bug in new python extension update in vscode. You can try to setup older version, it works for me.

Yup. Installing a previous version works

also installed previous version which works perfectly. Thank you

@evgYakim Excellent my friend. Thanks for helping.


After all test behaviour with select and exclude

Hello, the AFTER_ALL tests behavior is not considering the select and exclude settings defined in the rendering configuration.
Test Behavior: After All
Load Method: DBT ls
Execuction mode: Kubernetes
Test indirect selection: Buildable

Thanks for reporting this issue, @ndrluis! Would you like to help us fix this?

Hello @tatiana, I am interested, but I need help understanding where I should make this correction.
I am studying the code, and I'm not sure if I got it right. But I believe I would need to pass the select and exclude values to the build_airflow_graph function and then to create_test_task_metadata when TestBehavior.AFTER_ALL is used.
Within create_test_task_metadata, I would check if the load mode is DBT_LS and pass the values to the select and exclude arguments.
Is it something along these lines?

Hello @tatiana, I am interested, but I need help understanding where I should make this correction.
I am studying the code, and I'm not sure if I got it right. But I believe I would need to pass the select and exclude values to the build_airflow_graph function and then to create_test_task_metadata when TestBehavior.AFTER_ALL is used.

Yes, I believe we could pass render_config as an argument of build_airflow_graph and expose select/exclude in create_test_task_metadata, as you suggested.

Within create_test_task_metadata, I would check if the load mode is DBT_LS and pass the values to the select and exclude arguments.

I don't think we'd need to check if it is DBT_LS. Why do you think this would be necessary? This issue happens for other LoadModes as well, and it makes sense for the test task craeted to only run tests for the subset of tasks filtered using RenderConfig.

I have the same issue.


Handle connection refused error

Hello, I'm trying to handle a connection refused error, if I try to connect to a server that doesn't exist, my app crash.
I've try something like this:
try
{
    var client = net.createConnection({ port: 49500, host: "<IP_ADDRESS>",timeout:1000 })
}
catch (error)
{
    console.log(error)
}

and also:
 var client = net.createConnection({ port: 49500, host: "<IP_ADDRESS>",timeout:1000 })
 .then(val =>
{
    console.log("connected")
}
).catch(error => {
    console.log(error)
})

But the App still continue to crash with the Connection Refused Exception

Is it solved？

Is it solved？

Hello developers, I have just recently used the react-native TCP and I encountered with connection refused error while creating the connection if the server is closed, how I handled it, I used net.connect() method rather that create connection, then error event handle the connection refused error, I am pasting the code here so you can see it, Thank you...
const client = await net.connect(5000,data[i].ipdata_ip,()=>{
client.write(JSON.stringify({
tableNum: tab,
time: Moment(new Date()).format('MM-DD-YYYY hh:mm:ss A')
}));
client.destroy()
        })
        client.on('error',(error)=>{
          alert(`Configuration kitchen error at screen ${data[i].ipdata_ip}`)
        })

Hello developers, I have just recently used the react-native TCP and I encountered with connection refused error while creating the connection if the server is closed, how I handled it, I used net.connect() method rather that create connection, then error event handle the connection refused error, I am pasting the code here so you can see it, Thank you...
`const client = await net.connect(5000,data[i].ipdata_ip,()=>{
client.write(JSON.stringify({
tableNum: tab,
time: Moment(new Date()).format('MM-DD-YYYY hh:mm:ss A')
}));
client.destroy()
})
client.on('error',(error)=>{
  alert(Configuration kitchen error at screen ${data[i].ipdata_ip})
})`


Drop minimum .NET Framework to 4.0

Should into Windows 8.0 for sure now. If it somehow STILL doesn't, well...at least we'll know something else was amiss all along.

Side note: I can build binaries for release, if needed.

Thanks, I didn't think the version number was the problem, as the Microsoft documentation said 4.6 is available for 8.0.
As I fixed the problem, I will upload the binaries soon.

Merged. Could you try https://github.com/Ingan121/BasicThemer2/releases/tag/v0.5.2-3?

@Ingan121 Okay, this just got...interesting.
So I had my friend who's been testing against Windows 8.0 for me test https://cdn.discordapp.com/attachments/794213473508786188/857421639146143745/BasicThemer2-net40.exe out, and...

however, prior to opening this PR, I'd compiled from this branch myself, so after your binaries did the thing shown above, I had him test my binaries, and...

So I suspect that either the relevant change was somehow left out of this PR, or (perhaps more likely) something is still amiss on your end. Maybe try completely re-cloning your master branch and compile that?

I just did what you said, could you test this one?
BasicThemer2-v0.5.2-Release-4.zip
One weird thing is that doing this thing to this binary as well as the third binary both shows .net version 4.0. - Screenshot
Finding 4.7.2 shows nothing in the both binaries.

I just did what you said, could you test this one?
BasicThemer2-v0.5.2-Release-4.zip
One weird thing is that doing this thing to this binary as well as the third binary both shows .net version 4.0. - Screenshot
Finding 4.7.2 shows nothing in the both binaries.

No luck, still wants 4.7.2 ...

I will reinstall VS and try building it again sometime as I got it removed while installing W11.

Just installed VS and rebuilt it: BasicThemer2-v0.5.2-Release+Debug-5.zip
Please try the debug version if the release version somehow still doesn't work.


Pushing to Expo for master branch

I currently have a project set up to use appr by deploying pull requests (like the docs say).
Is there a way to deploy my master branch to the master Expo branch when master is updated?
Thanks!

This could definitely be done. All you'd really need to do is to NOT run the post-deploy script and leave out the  "$CI_PULL_REQUEST" != "" or  "$TRAVIS_PULL_REQUEST" != "" in the CI configuration.
Alternatively, you can run exp login and exp publish as CI steps post-build without appr, since that's what appr does internally.

We should investigate how we can support building non-PR branches. This would already work today, if we didn't make assumptions around having a PR to push comments to.


chore(deps): upgrade minimist to 1.2.3

Description of changes
Upgrade minimist from 1.2.2 to 1.2.3
Pull request checklist

[x] PR title respects Conventional Commits (starts with fix:, feat:, etc, and is suitable for user-facing release notes)
[x] PR contains no breaking changes, OR description of both PR and final merge commit starts with BREAKING CHANGE:
[n/a] (if applicable) Addresses issue: #0000
[n/a] Added relevant unit tests for your changes
[n/a] Ran yarn precheckin
[n/a] Verified code coverage for the changes made

:tada: This PR is included in version 2.3.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Support context for datastore interfaces

Is your feature request related to a problem? Please describe.
The current upload interfaces does not support context.Context. For example, in the DataStore interface, the methods do not take context into account:
https://github.com/tus/tusd/blob/d23be46d7a236714c451601f87fe6bbab7776d45/datastore.go#L47-L52
What this means is we're not able to propagate things like request cancellation into the datastore. It seems useful to support this as context has become one of the standard concurrency patterns in golang.
Describe the solution you'd like
For interface methods and their implementations, instead of:
type DataStore interface {
 	NewUpload(info FileInfo) (id string, err error)
}

type DataStore interface {
 	NewUpload(ctx context.Context, info FileInfo) (id string, err error)
}

This would mean breaking existing APIs. If we're ok with this (might be ok since it's only internal API), this can be handled in the release notes. For example, another SDK (Firebase Admin SDK), made an API-breaking release just for supporting context: https://github.com/firebase/firebase-admin-go/releases/tag/v3.0.0
Describe alternatives you've considered
To make the API more backwards compatible, one might add context-aware interfaces alongside with existing interfaces:
type DataStore interface {
 	NewUpload(info FileInfo) (id string, err error)
}

type ContextAwareDataStore interface {
 	NewUpload(ctx context.Context, info FileInfo) (id string, err error)
}

But that might end up with more confusion.
The consumer could also make their own custom context-aware wrapper, but it'd be nicer if it were supported by the official implementation.
Can you provide help with implementing this feature?
Yes
Additional context
None

That's a good idea, thanks for bringing it up. Unfortuately, the DataStore API is not internal but public, so we cannot change it would a breaking API change. Implementing a context-aware interface looks like a high cost for little benefit.
Instead, I think this is a good candidate for the next major release of tusd where we are making multiple breaking changes. We currently have collected a list of ideas in #212 and already started implementing a few of them. I added your proposal to this list.

Thanks a lot for the quick reply. I think making it as part of v1 makes sense.

Perfect, I will leave this issue open as a reminder until the feature has been implemented in v1.


As a new user, I want to be walked through all the setup I need to do before being set loose in the app, so that I don't wind up with stuff to do later

There is some data we need to collect before users can use the site (where did you hear about us and their email contact). Some stuff is set up for vocabulary (choose an upstream collection, set phonetics preference).
Requirements:

As a new user visiting the site for the first time...
I see a popup that explains the site, where to find more information (FAQ), and the general code of conduct (eg. flagging terms), then gives me a "continue" button

[ ] Charlie needs to write this

I am asked to give my email address and say where I found the site
I can choose to either:

Go straight to minigames (exit wizard, or maybe a "You're all set!" page)
Set up my daily vocab lessons

(if I choose to set up daily lessons) I am asked to choose a phonetics preference and pick my first collection

I see a live demo below to see what each preference means (ie. render syllabary and phonetics for some card, given the user's selected preference in the form -- just syllabary, syllabary and simple phonetics, syllabary and rich phonetics)
Each collection is listed with its description, credits, and the total number of terms.

Cherokee Language Lessons is not listed

(if I do not choose to set up daily lessons) The vocab/lessons section of the app is replaced with a "Set up daily vocab lessons" button, which will send me to the "if I choose to set up daily lessons" flow.

[ ] Errors for email in use, etc.
[ ] Sign in / Create account with Google
[ ] Forgot password page
[ ] Upgrading accounts


Beautify man pages

[ ] Beautify those ascii tree diagram
[ ] Words to avoid in those articles
[ ] Misc fixes / forgotten updates

+1 for words to avoid and I am sure I can think of more things.
One more thing I am trying actively to avoid is using adverbs.
As for the ascii tree diagram, I recall my man would not display unicode characters, so I desisted.

As for the ascii tree diagram, I recall my man would not display unicode characters, so I desisted.

Of course, I always know that and I will make that my first policy, I just want to say that the ascii tree diagram still looks a bit ugly and some part needs some fix.

One more thing I am trying actively to avoid is using adverbs.

I am not sure what you mean here, one more thing that I am doing is to make those sentences as short as possible as long as the original meanings remain.

Cool. Go for it.

@pickfire Now that Fisherman API is essentially locked, I will go over the documentation and rewrite and fix as much as possible too. In addition, I will simplify the Wiki, re-shoot gifs with the current Fisherman and consolidate as much as possible in preparation for 1.0.

I need to go through all the docs, give me some time. At the moment, you can work on the wiki.

@pickfire If you haven't started, then leave that up to me, and you can review it after I push the new version. I will be doing this from now, over the next few hours. I would love if you can fix anything after that.

I have started since last night, assign this to me and the other parallel thing (haven't started).

Well, the thing is, I will remove some of the docs and consolidate them into one or two documents. I wrote too much docs to begin with.


History navigation back and forward is not working with iframe.

When you change an iframe's src via javascript, sammy adds buggy history changes to the browsers navigation.
Ex:
$("iframe").attr("src","");

I also has the same problem.


Fauxmo ESP8266 Not recognized as a Bulb,etc.

Original report by Cole Rabe (Bitbucket: [Cole Rabe](https://bitbucket.org/Cole Rabe), ).

When I use the example sketches, the first thing that I have noticed is that I am not able to add more than 3 devices and then have the amazon echo discover them. Both example sketchs for me have this issue. The bigger issue for me is that when I do get the amazon echo to discover my devices (again, only 3), they do not show up as lights. Do I need any special smart home skill to make this work, or is there anything I need to modify code-wise to allow the echo to detect the device type? While my code is not the example sketch, it should theoretically work the same. Let me know however if there is any changes in my code that would need to be made to solve my issue!

Thanks!

Board: ESP8266
Echo Model: Amazon Echo Dot Version 3
Library Version: v3.1.1
Code:
#include <Arduino.h>
#include <ESPAsyncTCP.h>
#include <ESP8266WiFi.h>
#include <fauxmoESP.h>
#define WIFI_SSID "Wireless" // Please Enter you Wifi name here
#define WIFI_PASS "7817293793" // Enter password here
#define SERIAL_BAUDRATE     115200
fauxmoESP fauxmo;
void setup() {
    WiFi.hostname("MyESP8266");
    Serial.begin(SERIAL_BAUDRATE);
    Serial.println("~~~Basement TV LED Driver~~~");
     pinMode(D0, OUTPUT);
     pinMode(D1, OUTPUT);
     pinMode(D2, OUTPUT);
    Serial.println("Connecting to Network:");
    if (connectWifi()) {
    // Setup fauxmo
     fauxmo.setPort(80);
     fauxmo.enable(true);
     fauxmo.addDevice("Basement TV LED Red");
     fauxmo.addDevice("Basement TV LED Green");
     fauxmo.addDevice("Basement TV LED Blue");
  }
   fauxmo.onSetState([](unsigned char device_id, const char * device_name, bool state, unsigned char value) {
        Serial.printf("[MAIN] Device #%d (%s) state: %s value: %d\n", device_id, device_name, state ? "ON" : "OFF", value);

        if (strcmp(device_name, "Basement TV LED Red")==0) {
           if(state){
            digitalWrite(D0, HIGH);
            }
            else {
            digitalWrite(D0, LOW);
              }
}
       if (strcmp(device_name, "Basement LED Green")==0) {
           if(state){
            digitalWrite(D1, HIGH);
              }
            else {
            digitalWrite(D1, LOW);
              }
}
      if (strcmp(device_name, "Basement LED Blue")==0) {
           if(state){
            digitalWrite(D2, HIGH);
            }
            else {
            digitalWrite(D2, LOW);
              }
}
}
);
}
void loop() {
    fauxmo.handle();
}
boolean connectWifi() {
    WiFi.mode(WIFI_STA);
    WiFi.begin(WIFI_SSID, WIFI_PASS);
    while (WiFi.status() != WL_CONNECTED) {
        Serial.print(".");
        delay(100);
        }
    Serial.println(".......");
    Serial.println("WiFi Connected....IP Address: ");
    Serial.print(WiFi.localIP());
  return true;
}

I've been finding recently that when I add the devices, alexa adds them fine, but they show up in the app as a generic device as opposed to a light. It still functions as a light when I give voice commands, however.
Also, if I create a group in the Alexa app, and add device(s) to it, I can turn it off and on through the app (no dimming though). I'll be looking into this more. (Using Echo Dot Gen3).
for only being able to ad three devices - can you post the serial output?
Thanks
Paul

I've been confronted with the same problem. I recognized, that the alexa app is showing the fauxmo devices as type:"other" ("sonstige" in german, don't know what the exact naming in the english version is). So I started looking at FAUXMO_DEVICE_JSON_TEMPLATE[], where the type is declaired. I don't have any phillips bulbs but I recognized LTC007 is very old. Lacking of an example from Phillips I researched what type my ikea bulbs are exposing and exchanged the JSON to the following:
PROGMEM const char FAUXMO_DEVICE_JSON_TEMPLATE[] = "{"     "\"type\":\"Dimmable light\","     "\"name\":\"%s\","     "\"uniqueid\":\"%s%06d\","     "\"modelid\":\"TRADFRI_bulb_E14_W_op_ch_400lm\","     "\"state\":{"         "\"on\":%s,\"bri\":%d,\"xy\":[0,0],\"reachable\": true"     "},"     "\"capabilities\":{"         "\"certified\":false,"         "\"streaming\":{\"renderer\":true,\"proxy\":false}"     "},"     "\"swversion\":\"<IP_ADDRESS>69\"" "}";
From now on new devices are discovered as a light by the app and I can use the Alexa app for dimming and switching manually.
I can even replace the modelid with "fauxmo" and the discovery works as expected. The type property seems to be the counting one.

Awesome! Thanks @StoeSeppl !
I had played around with the modelid some with no success, but simply changing the type made it work!
A bit of a further test revealed that simply fixing the case of the type was the real key (instead of Extended Color Light change it to Extended color light).
I committed the change to master, but I'm still working on it. This gave me the one step I needed to get colours working.

Fixed in 0812901f0a564080a94241c6b9cc82e160de2f20

Fixed in 0812901f0a564080a94241c6b9cc82e160de2f20


Use explicit template instead of 'gcc'

cppcheck 1.84 changed the meaning of --template=gcc to add a second line of output that confuses the linter plugin, so explicitly request the old format
see https://github.com/danmar/cppcheck/commit/f058d9ad083a6111e9339b4b3506c5da7db579e0 for the details of that change

I just tested with cppcheck 1.87.
This is what the current command does:
$ cppcheck --template=gcc --inline-suppr --quiet addons/test/*
addons/test/misra2012_rules_dummy_ascii.txt:5:15: warning: syntax error [syntaxError]

              ^
addons/test/misra2012_rules_dummy_utf8.txt:3:0: warning: The code contains unhandled character(s) (character code=226). Neither unicode nor extended ascii is supported. [syntaxError]
Text of rule 1.1, utf8 test: ∑
^
addons/test/misra2012_rules_dummy_windows1250.txt:3:0: warning: The code contains unhandled character(s) (character code=146). Neither unicode nor extended ascii is supported. [syntaxError]
Text of rule 1.1, windows1250 test: �
^
addons/test/suppressions.txt:3:6: warning: syntax error [syntaxError]

     ^

This is what the proposed pull-request does:
$ cppcheck --template="{file}:{line}: {severity}: {message}" --inline-suppr --quiet addons/test/*
addons/test/misra2012_rules_dummy_ascii.txt:5: error: syntax error
addons/test/misra2012_rules_dummy_utf8.txt:3: error: The code contains unhandled character(s) (character code=226). Neither unicode nor extended ascii is supported.
addons/test/misra2012_rules_dummy_windows1250.txt:3: error: The code contains unhandled character(s) (character code=146). Neither unicode nor extended ascii is supported.
addons/test/suppressions.txt:3: error: syntax error

@unwiredben it seems Travis-CI failed because the new cmd is too long [1].
Can you change it to read like this so the patch can pass the 120 character limit?
cmd = (
    'cppcheck',
    '--template={file}:{line}: {severity}: {message}',
    '--inline-suppr',
    '--quiet',
    '${args}',
    '${file}'
)

[1] https://travis-ci.org/SublimeLinter/SublimeLinter-cppcheck/builds/516703050

Hi! Please also output the 'column'.
Ideally, we would also capture 'id' as 'code'.

I think that looks 👌!
@cixtor wanna 🚢 ?


GitHub Logo Size

Issue Template
Prerequisites
Please answer the following questions for yourself before submitting an issue. YOU MAY DELETE THE PREREQUISITES SECTION.

[ ] I am running the latest version
[ ] I checked the documentation and found no answer
[x] I checked to make sure that this issue has not already been filed
[ ] I'm reporting the issue to the correct repository (for multi-repository projects)

Context
Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

Firmware Version:
Operating System:
SDK version:
Toolchain version:

Expected Behavior
The logo will be increased in size
Current Behavior
The size of logo is small
Steps to Reproduce
Please provide detailed steps for reproducing the issue.

The logo will be increased in size using css

Please assign this issue to me under GSSOC'23

this is a too easy issue to be assigned for gssoc


Cannot set multiple lineWidths to xAxes gridLines

I'm trying to set a lineWidth for each x value. I tried to do as described on the docs (e.g. [1,1,1,1,1,40,1,1,1,1,1,1]), but doesn't work, making all gridLines for the xAxis disappear.
The same approach was used to change the color for each x value and it works perfectly.
Here what I tried:
options={{
    responsive: false,
    legend: {
        display: false,
    },
    maintainAspectRatio: false,
    scales: {
        xAxes: [{
            stacked: true,
            gridLines: {
                display: true,
                drawBorder: false,
                offsetGridLines: false
            }
        }],
        yAxes: [{
            display: true,
            stacked: true,
            ticks: {
                beginAtZero: true,
                max: datas.options.max,
                stepSize: datas.options.stepSize
            }
        }]
    }
}}

Please refer to the Chart.js documentation


Fix issue when building without GPL.

If -DLICENSE_GPL_SHOGUN=OFF -DUSE_SVMLIGHT=OFF are provided then the cmake procedure fails because it does not find some SVM meta examples (which are non-existing).

Sooo, shall we merge this? ;)

I think we should find out where those lines initially came from and then either confirm that those files were deleted (which commit, why), or find out the new name and then replace the filenames. We had renamed a few examples a while ago and I have the feeling that those for part of that.

Okay, so, I did again a bit of research and I found the real problem. Those two lines were added here https://github.com/shogun-toolbox/shogun/commit/9e5199bb45bef4196f20a14c00426fe400053677 when we merged this PR #4361. The purpose of the PR was to port two undocumented examples (classifier_svmlin.py and classifier_svmsgd.py) to the meta example system. However, the PR lacks the "translated" .sg files for the meta examples :/ therefore, those two files never existed.
The solution here would be to add again those two files (since they are actually useful).

Hey @vinx13! Do you remember those two files (svmlin.sg and svmsgd.sg)? If yes, do you happen to still have them somewhere? ;)

@geektoni Seems that I didn't include that two files in the pr, I need some time to investigate it

Thanks for sorting this out guys!!


Docs reference "Messaging Service" in places where "Dashboard Service" or "Pipeline Service" are more appropriate

There are a few places in the documentation where Messaging Service is referenced when it should be Dashboard or Pipeline.  I assume this is due to a copy-paste.

https://docs.open-metadata.org/openmetadata-apis/schemas/entities/dashboardservice

Thank you for the contribution. Can you please open a PR we will merge it in https://github.com/open-metadata/OpenMetadata/pulls


LABS-2037: Improve logging for SQS in Kibana and add maat_reference

What
This PR adds support for additional params for the SQS logging on Kibana.
Log maat_reference to the log on publish messages.
Link to story

This is successfully tested. We can search for maat ref in the Kibana logs and see also the SQS messages relevant to the meat ref. That makes debugging much more straightforward.
Screenshot Example on Staging:


i want to add "dice rolle" extension

Is your feature request related to a problem? Please describe.
I'm always frustrated when I can't find a convenient and easy-to-use dice roller extension for my game projects.
Describe the solution you'd like
I want to add a user-friendly and visually appealing dice roller extension to my existing game.
Describe alternatives you've considered
I've considered using third-party dice roller tools or creating a basic dice roller without advanced features.
Additional context
The extension should have animations and fit seamlessly with the existing game design.
What problem is this feature trying to solve?
It aims to provide a convenient and integrated dice rolling solution within the game.

hey @sanjay-kv assign issue to me as GSSOC'24 contibutor


Problem storing data on column with name "description"

Hi everyone,
I found this issue today. When I try to store data on a column "description", it doesn't work. But if I use any other column name it work's correctly (ex: "descriptionx"):
var database = new Dexie("webapp");
database.version(3).stores({
  "appointment": "id,firstname,lastname,phone,email,start,end,description"
});
database.open();
database["appointment"].put({
                        id: "123",
                        firstname: "lucas",
                        lastname: "moy",
                        description: "descripcion",
                        phone: "234234234",
                        email<EMAIL_ADDRESS>                        start: moment().format('YYYY-MM-DD'),
                        end: moment().format('YYYY-MM-DD')
                    });

I really like Dexie. Good job guys 👍

This really surprise me since there's no such built in limitation. Please put your repro into a codesandbox or jsfiddle or similar and paste it's link into this issue. If you're still able to reproduce the issue, i will easily find what it could be.

This really surprise me since there's no such built in limitation. Please put your repro into a codesandbox or jsfiddle or similar and paste it's link into this issue. If you're still able to reproduce the issue, i will easily find what it could be.

Sorry David, it was my mistake. It works fine, I'll be more careful before reporting a bug. Have a nice day.

Sorry David, it was my mistake. It works fine, I'll be more careful before reporting a bug. Have a nice day.


CRUD Schema support for BaseSessionContext

Which issue does this PR close?
Closes #391

PR #390 should be merged first


Testing workshop

Organize, plan and facilitate a testing workshop
Workshop Outline

Why test?

Align code with business goals and needs
Code documentation
Prevent regression
Confidence when refactoring
Make sure our assumptions are correct
Project and team scaling

Behavior Driven Design

User Stories - interactive (bakery)

As a ... [role]
I want ... [feature or activity]
So that ... [outcome]

Scenarios - interactive (bakery)

Given ... [initial setup]
When ... [event or action]
then ... [expected outcome]

Types of Tests - system diagram

Unit
Integration
Functional (Browser)

Tests are functions
Test Doubles
Testing sequence

Setup
Execute
Verify
Teardown

Jasmine

beforeAll()
describe()

it()

afterAll()

Browser Testing

Casper & Slimer

Resources

End-to-end testing with Meteor
Introducing BDD by Dan North
JavaScript testing with Jasmine - video series

Bakery examples:
spec.js
  it('should bake bread', function () {
    expect(bakeBread()).toEqual('Fresh baked bread!');
  })

  it ('should bake customer favorite bread', function () {
    expect(bakeCustomerFavoriteBread('rye')).toEqual('Fresh baked rye bread!');
  })
})

source.js
var bakeBread = function (breadType) {
  if (breadType) {
    return 'Fresh baked ' + breadType + ' bread!'
  } else {
    return 'Fresh baked bread!';
  }
};

var bakeCustomerFavoriteBread = function (favoriteBread) {
  var bakedBread = bakeBread(favoriteBread);

  return bakedBread;
}


 But if the user misspells the filter type, there is no indication to the user what happened. I would rather panic, when the filter type cannot be recognized so that the user sees exactly what's happening.
Also, the no-filter option doesn't seem to work. No messages came through when selected.
Last, but not least, it would be cool to have the -n option as a shortcut for --no-filter.
What do you think?

Were you asking me or @wfernandes  ?

@erikjasiak I'm pretty sure it was directed to me. Will discuss my thoughts with since he's standing right here.


Fix parsing of input and return empty solution

This PR enables the parsing of instances and it also returns the empty solution.
Follow-up PR will add some very simple single-order solving.

I have read the CLA Document and I hereby sign the CLA


Potential bug with first order MAML using only higher, setting track_higher_grads = False leads to .grad field to not be populated and be None, is that a bug?

I was reading this gitissue:
https://github.com/facebookresearch/higher/issues/63
and as far as I understand setting track_higher_grads = False should implement foMAML. But in my experience it actually leads to the .grad field to be None. Is this a bug?
This can be easily reproducible with the omniglot example provided by switching track_higher_grads = False.
Here is the code:
#!/usr/bin/env python3
#
# Copyright (c) Facebook, Inc. and its affiliates.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
This example shows how to use higher to do Model Agnostic Meta Learning (MAML)
for few-shot Omniglot classification.
For more details see the original MAML paper:
https://arxiv.org/abs/1703.03400

This code has been modified from Jackie Loong's PyTorch MAML implementation:
https://github.com/dragen1860/MAML-Pytorch/blob/master/omniglot_train.py

Our MAML++ fork and experiments are available at:
https://github.com/bamos/HowToTrainYourMAMLPytorch
"""

import argparse
import time
import typing

import pandas as pd
import numpy as np
import matplotlib as mpl

mpl.use('Agg')
import matplotlib.pyplot as plt

plt.style.use('bmh')

import torch
from torch import nn
import torch.nn.functional as F
import torch.optim as optim

import higher

from support.omniglot_loaders import OmniglotNShot

def main():
    argparser = argparse.ArgumentParser()
    argparser.add_argument('--n_way', type=int, help='n way', default=5)
    argparser.add_argument(
        '--k_spt', type=int, help='k shot for support set', default=5)
    argparser.add_argument(
        '--k_qry', type=int, help='k shot for query set', default=15)
    argparser.add_argument(
        '--task_num',
        type=int,
        help='meta batch size, namely task num',
        default=32)
    argparser.add_argument('--seed', type=int, help='random seed', default=1)
    args = argparser.parse_args()

    torch.manual_seed(args.seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed_all(args.seed)
    np.random.seed(args.seed)

    # Set up the Omniglot loader.
    # device = torch.device('cuda')
    # from uutils.torch_uu import get_device
    # device = get_device()
    device =  torch.device(f"cuda:{gpu_idx}" if torch.cuda.is_available() else "cpu")
    db = OmniglotNShot(
        '/tmp/omniglot-data',
        batchsz=args.task_num,
        n_way=args.n_way,
        k_shot=args.k_spt,
        k_query=args.k_qry,
        imgsz=28,
        device=device,
    )

    # Create a vanilla PyTorch neural network that will be
    # automatically monkey-patched by higher later.
    # Before higher, models could *not* be created like this
    # and the parameters needed to be manually updated and copied
    # for the updates.
    net = nn.Sequential(
        nn.Conv2d(1, 64, 3),
        nn.BatchNorm2d(64, momentum=1, affine=True),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(2, 2),
        nn.Conv2d(64, 64, 3),
        nn.BatchNorm2d(64, momentum=1, affine=True),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(2, 2),
        nn.Conv2d(64, 64, 3),
        nn.BatchNorm2d(64, momentum=1, affine=True),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(2, 2),
        Flatten(),
        nn.Linear(64, args.n_way)).to(device)

    # We will use Adam to (meta-)optimize the initial parameters
    # to be adapted.
    meta_opt = optim.Adam(net.parameters(), lr=1e-3)

    log = []
    for epoch in range(100):
        train(db, net, device, meta_opt, epoch, log)
        test(db, net, device, epoch, log)
        # plot(log)

def train(db, net, device, meta_opt, epoch, log):
    net.train()
    n_train_iter = db.x_train.shape[0] // db.batchsz

    for batch_idx in range(n_train_iter):
        start_time = time.time()
        # Sample a batch of support and query images and labels.
        x_spt, y_spt, x_qry, y_qry = db.next()

        task_num, setsz, c_, h, w = x_spt.size()
        querysz = x_qry.size(1)

        # TODO: Maybe pull this out into a separate module so it
        # doesn't have to be duplicated between `train` and `test`?

        # Initialize the inner optimizer to adapt the parameters to
        # the support set.
        n_inner_iter = 5
        inner_opt = torch.optim.SGD(net.parameters(), lr=1e-1)

        qry_losses = []
        qry_accs = []
        meta_opt.zero_grad()
        for i in range(task_num):
            with higher.innerloop_ctx(
                    net, inner_opt, copy_initial_weights=False,
                    # track_higher_grads=True,
                    track_higher_grads=False,
            ) as (fnet, diffopt):
                # Optimize the likelihood of the support set by taking
                # gradient steps w.r.t. the model's parameters.
                # This adapts the model's meta-parameters to the task.
                # higher is able to automatically keep copies of
                # your network's parameters as they are being updated.
                for _ in range(n_inner_iter):
                    spt_logits = fnet(x_spt[i])
                    spt_loss = F.cross_entropy(spt_logits, y_spt[i])
                    diffopt.step(spt_loss)

                # The final set of adapted parameters will induce some
                # final loss and accuracy on the query dataset.
                # These will be used to update the model's meta-parameters.
                qry_logits = fnet(x_qry[i])
                qry_loss = F.cross_entropy(qry_logits, y_qry[i])
                qry_losses.append(qry_loss.detach())
                qry_acc = (qry_logits.argmax(
                    dim=1) == y_qry[i]).sum().item() / querysz
                qry_accs.append(qry_acc)

                # Update the model's meta-parameters to optimize the query
                # losses across all of the tasks sampled in this batch.
                # This unrolls through the gradient steps.
                qry_loss.backward()

        assert meta_opt.param_groups[0]['params'][0].grad is not None
        meta_opt.step()
        qry_losses = sum(qry_losses) / task_num
        qry_accs = 100. * sum(qry_accs) / task_num
        i = epoch + float(batch_idx) / n_train_iter
        iter_time = time.time() - start_time
        if batch_idx % 4 == 0:
            print(
                f'[Epoch {i:.2f}] Train Loss: {qry_losses:.2f} | Acc: {qry_accs:.2f} | Time: {iter_time:.2f}'
            )

        log.append({
            'epoch': i,
            'loss': qry_losses,
            'acc': qry_accs,
            'mode': 'train',
            'time': time.time(),
        })

def test(db, net, device, epoch, log):
    # Crucially in our testing procedure here, we do *not* fine-tune
    # the model during testing for simplicity.
    # Most research papers using MAML for this task do an extra
    # stage of fine-tuning here that should be added if you are
    # adapting this code for research.
    net.train()
    n_test_iter = db.x_test.shape[0] // db.batchsz

    qry_losses = []
    qry_accs = []

    for batch_idx in range(n_test_iter):
        x_spt, y_spt, x_qry, y_qry = db.next('test')

        task_num, setsz, c_, h, w = x_spt.size()
        querysz = x_qry.size(1)

        # doesn't have to be duplicated between `train` and `test`?
        n_inner_iter = 5
        inner_opt = torch.optim.SGD(net.parameters(), lr=1e-1)

        for i in range(task_num):
            with higher.innerloop_ctx(net, inner_opt, track_higher_grads=False) as (fnet, diffopt):
                # Optimize the likelihood of the support set by taking
                # gradient steps w.r.t. the model's parameters.
                # This adapts the model's meta-parameters to the task.
                for _ in range(n_inner_iter):
                    spt_logits = fnet(x_spt[i])
                    spt_loss = F.cross_entropy(spt_logits, y_spt[i])
                    diffopt.step(spt_loss)

                # The query loss and acc induced by these parameters.
                qry_logits = fnet(x_qry[i]).detach()
                qry_loss = F.cross_entropy(
                    qry_logits, y_qry[i], reduction='none')
                qry_losses.append(qry_loss.detach())
                qry_accs.append(
                    (qry_logits.argmax(dim=1) == y_qry[i]).detach())

    qry_losses = torch.cat(qry_losses).mean().item()
    qry_accs = 100. * torch.cat(qry_accs).float().mean().item()
    print(
        f'[Epoch {epoch + 1:.2f}] Test Loss: {qry_losses:.2f} | Acc: {qry_accs:.2f}'
    )
    log.append({
        'epoch': epoch + 1,
        'loss': qry_losses,
        'acc': qry_accs,
        'mode': 'test',
        'time': time.time(),
    })

def plot(log):
    # Generally you should pull your plotting code out of your training
    # script but we are doing it here for brevity.
    df = pd.DataFrame(log)

    fig, ax = plt.subplots(figsize=(6, 4))
    train_df = df[df['mode'] == 'train']
    test_df = df[df['mode'] == 'test']
    ax.plot(train_df['epoch'], train_df['acc'], label='Train')
    ax.plot(test_df['epoch'], test_df['acc'], label='Test')
    ax.set_xlabel('Epoch')
    ax.set_ylabel('Accuracy')
    ax.set_ylim(70, 100)
    fig.legend(ncol=2, loc='lower right')
    fig.tight_layout()
    fname = 'maml-accs.png'
    print(f'--- Plotting accuracy to {fname}')
    fig.savefig(fname)
    plt.close(fig)

# Won't need this after this PR is merged in:
# https://github.com/pytorch/pytorch/pull/22245
class Flatten(nn.Module):
    def forward(self, input):
        return input.view(input.size(0), -1)

if __name__ == '__main__':
    main()

I have it live all in one place here too: https://github.com/brando90/ultimate-utils/tree/master/tutorials_for_myself/higher_uu

related:

https://stackoverflow.com/questions/70961541/what-is-the-official-implementation-of-first-order-maml-using-the-higher-pytorch
https://github.com/facebookresearch/higher/issues?q=is%3Aissue+first+order+maml+is%3Aclosed
https://github.com/facebookresearch/higher/issues/63
https://github.com/facebookresearch/higher/issues/128

@egrefen apologies for the direct ping. But I was wondering, should setting tracking_higher_grads=False implement first order MAML? I get that the .grad = None which seemed like a bug afaik from the other discussion.
Can someone confirm or route this to the right person?

@brando90 my answer here might help. It has the benefit that it should also still propagate gradients to learning rates, if you want to do foMAML with learned learning rates. Overall, I do agree that this behavior is a bit unintuitive.

@eric-mitchell  Funny, you mentioned this to me a few months back but ended up having to return to this and see why it's not populating the grad fields. Will check your link since I believe I need it.

Took me some effort to create a self contained reproducible script of this issue so I'm sharing it:
import os
from pathlib import Path

import torch
import torch.nn as nn
import torch.nn.functional as F
from tqdm import tqdm
import logging

from collections import OrderedDict

import higher  # tested with higher v0.2

from torchmeta.datasets.helpers import omniglot
from torchmeta.utils.data import BatchMetaDataLoader

logger = logging.getLogger(__name__)

def conv3x3(in_channels, out_channels, **kwargs):
    return nn.Sequential(
        nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1, **kwargs),
        nn.BatchNorm2d(out_channels, momentum=1., track_running_stats=False),
        nn.ReLU(),
        nn.MaxPool2d(2)
    )

class ConvolutionalNeuralNetwork(nn.Module):
    def __init__(self, in_channels, out_features, hidden_size=64):
        super(ConvolutionalNeuralNetwork, self).__init__()
        self.in_channels = in_channels
        self.out_features = out_features
        self.hidden_size = hidden_size

        self.features = nn.Sequential(
            conv3x3(in_channels, hidden_size),
            conv3x3(hidden_size, hidden_size),
            conv3x3(hidden_size, hidden_size),
            conv3x3(hidden_size, hidden_size)
        )

        self.classifier = nn.Linear(hidden_size, out_features)

    def forward(self, inputs, params=None):
        features = self.features(inputs)
        features = features.view((features.size(0), -1))
        logits = self.classifier(features)
        return logits

def get_accuracy(logits, targets):
    """Compute the accuracy (after adaptation) of MAML on the test/query points
    Parameters
    ----------
    logits : `torch.FloatTensor` instance
        Outputs/logits of the model on the query points. This tensor has shape
        `(num_examples, num_classes)`.
    targets : `torch.LongTensor` instance
        A tensor containing the targets of the query points. This tensor has
        shape `(num_examples,)`.
    Returns
    -------
    accuracy : `torch.FloatTensor` instance
        Mean accuracy on the query points
    """
    _, predictions = torch.max(logits, dim=-1)
    return torch.mean(predictions.eq(targets).float())

def train(args):
    logger.warning('This script is an example to showcase the data-loading '
                   'features of Torchmeta in conjunction with using higher to '
                   'make models "unrollable" and optimizers differentiable, '
                   'and as such has been  very lightly tested.')

    dataset = omniglot(args.folder,
                       shots=args.num_shots,
                       ways=args.num_ways,
                       shuffle=True,
                       test_shots=15,
                       meta_train=True,
                       download=args.download,
                       )
    dataloader = BatchMetaDataLoader(dataset,
                                     batch_size=args.batch_size,
                                     shuffle=True,
                                     num_workers=args.num_workers)

    model = ConvolutionalNeuralNetwork(1,
                                       args.num_ways,
                                       hidden_size=args.hidden_size)
    model.to(device=args.device)
    model.train()
    inner_optimiser = torch.optim.SGD(model.parameters(), lr=args.step_size)
    meta_optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

    # Training loop
    with tqdm(dataloader, total=args.num_batches) as pbar:
        for batch_idx, batch in enumerate(pbar):
            model.zero_grad()

            train_inputs, train_targets = batch['train']
            train_inputs = train_inputs.to(device=args.device)
            train_targets = train_targets.to(device=args.device)

            test_inputs, test_targets = batch['test']
            test_inputs = test_inputs.to(device=args.device)
            test_targets = test_targets.to(device=args.device)

            outer_loss = torch.tensor(0., device=args.device)
            accuracy = torch.tensor(0., device=args.device)

            for task_idx, (train_input, train_target, test_input,
                           test_target) in enumerate(zip(train_inputs, train_targets,
                                                         test_inputs, test_targets)):
                # track_higher_grads = True
                track_higher_grads = False
                with higher.innerloop_ctx(model, inner_optimiser, track_higher_grads=track_higher_grads, copy_initial_weights=False) as (fmodel, diffopt):
                    train_logit = fmodel(train_input)
                    inner_loss = F.cross_entropy(train_logit, train_target)

                    diffopt.step(inner_loss)

                    test_logit = fmodel(test_input)
                    outer_loss += F.cross_entropy(test_logit, test_target)

                    # inspired by https://github.com/facebookresearch/higher/blob/15a247ac06cac0d22601322677daff0dcfff062e/examples/maml-omniglot.py#L165
                    # outer_loss = F.cross_entropy(test_logit, test_target)
                    # outer_loss.backward()

                    with torch.no_grad():
                        accuracy += get_accuracy(test_logit, test_target)

            outer_loss.div_(args.batch_size)
            accuracy.div_(args.batch_size)

            outer_loss.backward()
            # print(list(model.parameters()))
            # print(f"{meta_optimizer.param_groups[0]['params'] is list(model.parameters())}")
            print(f"{meta_optimizer.param_groups[0]['params'][0].grad is not None=}")
            print(f"{meta_optimizer.param_groups[0]['params'][0].grad.norm()}")
            assert meta_optimizer.param_groups[0]['params'][0].grad is not None
            meta_optimizer.step()

            pbar.set_postfix(accuracy='{0:.4f}'.format(accuracy.item()))
            if batch_idx >= args.num_batches:
                break

    # Save model
    if args.output_folder is not None:
        filename = os.path.join(args.output_folder, 'maml_omniglot_'
                                                    '{0}shot_{1}way.th'.format(args.num_shots, args.num_ways))
        with open(filename, 'wb') as f:
            state_dict = model.state_dict()
            torch.save(state_dict, f)

if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser('Model-Agnostic Meta-Learning (MAML)')

    parser.add_argument('--folder', type=str, default=Path('~/data/torchmeta_data').expanduser(),
                        help='Path to the folder the data is downloaded to.')
    parser.add_argument('--num-shots', type=int, default=5,
                        help='Number of examples per class (k in "k-shot", default: 5).')
    parser.add_argument('--num-ways', type=int, default=5,
                        help='Number of classes per task (N in "N-way", default: 5).')

    parser.add_argument('--step-size', type=float, default=0.4,
                        help='Step-size for the gradient step for adaptation (default: 0.4).')
    parser.add_argument('--hidden-size', type=int, default=64,
                        help='Number of channels for each convolutional layer (default: 64).')

    parser.add_argument('--output-folder', type=str, default=None,
                        help='Path to the output folder for saving the model (optional).')
    parser.add_argument('--batch-size', type=int, default=16,
                        help='Number of tasks in a mini-batch of tasks (default: 16).')
    parser.add_argument('--num-batches', type=int, default=100,
                        help='Number of batches the model is trained over (default: 100).')
    parser.add_argument('--num-workers', type=int, default=1,
                        help='Number of workers for data loading (default: 1).')
    parser.add_argument('--download', action='store_false',
                        help='Do not Download the Omniglot dataset in the data folder.')
    parser.add_argument('--use-cuda', action='store_true',
                        help='Use CUDA if available.')

    args = parser.parse_args()
    args.device = torch.device('cuda' if args.use_cuda
                                         and torch.cuda.is_available() else 'cpu')

    train(args)

based on this: https://github.com/tristandeleu/pytorch-meta/blob/master/examples/maml-higher/train.py


Typo/mistake in slide 44 lecture 5

In order to separate the economists' full names by the full stop, the command should (surely?) be separate( ... , sep = "\\."), since "." is a regex.

Good catch, thanks!