classloaders leaking memory and file locks through JarFiles

steps
scripted test running on Windows (e.g. appveyor)
build.sbt
libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test"
ivyLoggingLevel := UpdateLogging.Quiet
exportJars := true
fork := true

test
# this test is only relevant on Windows
# where stale references to the jars will
# block file deletion of the jars.
> test
$ exists target/scala-2.10/exportjars_2.10-0.1-SNAPSHOT.jar
$ exists target/scala-2.10/exportjars_2.10-0.1-SNAPSHOT-tests.jar
$ delete target/scala-2.10/exportjars_2.10-0.1-SNAPSHOT.jar
$ delete target/scala-2.10/exportjars_2.10-0.1-SNAPSHOT-tests.jar
$ absent target/scala-2.10/exportjars_2.10-0.1-SNAPSHOT.jar
$ absent target/scala-2.10/exportjars_2.10-0.1-SNAPSHOT-tests.jar

problem
There is a well-known bug in the JVM, that classloaders do not release their jars

http://management-platform.blogspot.co.uk/2009/01/classloaders-keeping-jar-files-open.html
https://github.com/ancoron/apache-cxf/blob/master/common/common/src/main/java/org/apache/cxf/common/logging/JDKBugHacks.java
http://stackoverflow.com/questions/7371277

this manifests as a memory leak on all platforms, but on Windows it is particularly bad because file handles mean a lock on the jars, which means they cannot be deleted, moved or changed.
A catastrophic consequence is that compilations silently fail (!!!) on Windows. This is not a corner case, it happens every single time reliably.
I experience this when I run tests when the junit-interface is loaded but after spending the evening reading the junit-interface code, I am pretty convinced that the bug can only be worked around at the level of the classloader, and sbt is the owner of that lifecycle.
expectation
no leaks, no file locks.
notes
Originally raised at https://github.com/sbt/junit-interface/issues/67
This escapes the workaround proposed in https://github.com/sbt/sbt/issues/1223 as classloader jars seem to get "lost" in the heap and are never reclaimed by explicit GC or runFinalizers.
The scala compiler is also doing a similar thing, as I observed in https://issues.scala-lang.org/browse/SI-9682

If this isn't an Elmo, it's at least an Ernie

Note that URLClassLoader has a public close() method in JDK7+: http://docs.oracle.com/javase/7/docs/technotes/guides/net/ClassLoader.html

cool, that simplifies the workaround... just call it reflectively. Good find! I was going to suggest changing the classloader.

Ok, I have a reproduction in the sbt codebase for this https://github.com/fommil/sbt/commits/fix-classloaders
It passes in Linux, but fails on Windows. Can somebody please add appveyor? I can set it up to only run this one test initially and then maybe we can look at expanding its scope to cover more tests.

I'm looking for classloaders in sbt and being a little perturbed by the fact that there is an entire package called classpath with many many implementations. Here are some suspicious locations that require further investigation. Every single one of these instantiations really needs a close() call when its done, which is somewhat terrifying:
compile/src/main/scala/sbt/compiler/AnalyzingCompiler.scala:115:11:      new URLClassLoader(Array(interfaceJar.toURI.toURL), dual)
util/classpath/src/main/scala/sbt/classpath/ClasspathUtilities.scala:17:76:  def toLoader(finder: PathFinder, parent: ClassLoader): ClassLoader = new URLClassLoader(finder.getURLs, parent)
util/classpath/src/main/scala/sbt/classpath/ClasspathUtilities.scala:20:74:  def toLoader(paths: Seq[File], parent: ClassLoader): ClassLoader = new URLClassLoader(Path.toURLs(paths), parent)
util/classpath/src/main/scala/sbt/classpath/ClasspathUtilities.scala:23:9:    new URLClassLoader(Path.toURLs(paths), parent) with RawResources { override def resources = resourceMap }
util/classpath/src/main/scala/sbt/classpath/ClasspathUtilities.scala:26:9:    new URLClassLoader(Path.toURLs(paths), parent) with RawResources with NativeCopyLoader {
util/classpath/src/main/scala/sbt/classpath/ClasspathUtilities.scala:30:14:        s"""|URLClassLoader with NativeCopyLoader with RawResources(
util/classpath/src/main/scala/sbt/classpath/ClasspathUtilities.scala:103:24:      val loader = new URLClassLoader(Path.toURLs(classpath))
util/classpath/src/main/scala/sbt/classpath/ClassLoaderCache.scala:36:24:      val loader = new URLClassLoader(files.map(_.toURI.toURL).toArray, commonParent)
util/classpath/src/main/scala/sbt/ScalaInstance.scala:145:9:    new URLClassLoader(jars.map(_.toURI.toURL).toArray[URL], launcher.topLoader)
main/src/main/scala/sbt/PluginManagement.scala:40:57:  final class PluginClassLoader(p: ClassLoader) extends URLClassLoader(Array(), p) {
main/actions/src/main/scala/sbt/compiler/Eval.scala:148:53:          val loader = (parent: ClassLoader) => new URLClassLoader(Array(back.toURI.toURL), parent)

for this specific bug, perhaps TestFramework could provide a hacky workaround.

the good news is that there are more toLoader methods than users.
compile/integration/src/main/scala/sbt/compiler/AggressiveCompile.scala:119:45:            val loader = ClasspathUtilities.toLoader(searchClasspath)
compile/integration/src/main/scala/sbt/compiler/javac/AnalyzingJavaCompiler.scala:62:39:      val loader = ClasspathUtilities.toLoader(searchClasspath)
scripted/plugin/src/main/scala/sbt/ScriptedPlugin.scala:32:39:      val loader = ClasspathUtilities.toLoader(classpath, scala.loader)
project/Scripted.scala:72:47:    val loader = classpath.ClasspathUtilities.toLoader(scriptedSbtClasspath.files, noJLine)
main/src/main/scala/sbt/Load.scala:830:30:          ClasspathUtilities.toLoader(data(defcp), parentLoader)
main/command/src/main/scala/sbt/BasicCommands.scala:129:54:      val loader = if (cp.isEmpty) parentLoader else toLoader(cp.map(f => new File(f)), parentLoader)

I have appveyor running off my branch https://ci.appveyor.com/project/fommil/sbt

how can I turn off this error temporarily for this one test? It's never going to pass on JDK6
java.lang.RuntimeException: javac version 1.7.0_79 may not be used to publish, it has to start with 1.6 due to javacVersionPrefix setting

I have a hack.

@fommil remove Release.javaVersionCheckSettings from https://github.com/sbt/sbt/blob/0.13/build.sbt#L36

Thanks, but we'll need a repeatable test. I've got it listening to an envvar.

got a repeatible test in CI and also locally with my VirtualBox Windows 7 image. I'll try throwing some rocks in some random directions tomorrow and hope one lands on the bug.

fundamentally, ClassLoader should be treated like a Closeable resource. That looks like it would be a mad crazy refactor.

minimised to definedTests (not just test)

well, some progress. By explicitly closing the root of the ClasspathFilter I'm now getting failures to lookup the JUnitRunner when running the tests, but the definedTests piece now passes!
I think all I need to do now is to have a classloader that allows the jars to be closed, but still work when somebody tries to access them again.
I'm completely confused why this only breaks for junit tests and not scalatest tests. @bvenners (sorry to drag you into this) can you think of anything obvious? How does the scalatest TestFramework interact with the classloader?

oh, Java.  We need to get in at this private member
    private WeakHashMap<Closeable,Void>
        closeables = new WeakHashMap<>();

(we must synchronize on it) and go close all the JarFile instances. Then I think the classloader can continue to operate.
I'll try doing that, but I don't have much hope because reflection doesn't tend to work on these core classes (the JVM lies)

as expected, getDeclaredField gives a NoSuchFieldException :cry:
it might be possible to subclass URLClassLoader to get access.

aha! I was using .getClass and you've already subclassed it. This is looking good.

I won't be able to look at this for a week, so if anybody wants to pick up my branch, I got as far as reflectively looking up the closeables field and finding that it was empty. Which was somewhat unexpected.

I'm dedicating today to this and coming up with a workaround.

the classloader I found is only holding references to sbt-launch.jar and jre\lib\resources.jar, but at least the reflective approach to closing the cached jars in the URLClassLoader is working OK.

even running an explicit GC / runFinalizers / GC after detectTests I can still see the jars in the finalizer queue. Either the finalizer thread is getting stuck, or it just can't reach the objects anymore to finalize them. I can see the references in a heap dump. This is very very weird.

I'm back to thinking that maybe this could be fixed in junit-interface

ok, I've got hold of the URLClassLoader with the reference to the jars! However, it is claiming that it doesn't have JarFile references, so I'm not 100% sure what's going on.

lovely, the URLClassPath could also have the JarFiles http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/sun/misc/URLClassPath.java

W0000000000000000TTTTTT!!!!! Through evil manipulation of the deepest, darkest, corners of sun.misc private final fields, I have slayed the beast!

Given that my fix is insane, I can't see this ever making it into sbt proper. I'll package this up as a standalone workaround... which will involve using reflection to read private fields in sbt's classloaders.
// Copyright (C) 2016 Sam Halliday
// Licence: http://www.apache.org/licenses/LICENSE-2.0
package sbt

import scala.annotation.tailrec

object CloseClassloader {
  def close(cl: ClassLoader, log: Logger): Unit = {
    cl match {
      case u: java.net.URLClassLoader =>
        // do not use .getClass here as it is likely subclassed
        val closeablesField = classOf[java.net.URLClassLoader].getDeclaredField("closeables")

        log.info(s"classloader = ${u.getURLs.toList}")

        closeablesField.setAccessible(true)
        val closeables = closeablesField.get(u).asInstanceOf[java.util.WeakHashMap[java.io.Closeable, Void]]

        closeables synchronized {
          val it = closeables.keySet().iterator()
          while (it.hasNext()) {
            val closeable = it.next()
            closeable match {
              case jar: java.util.jar.JarFile =>
                log.warn(s"closing ${jar.getName}")
              case _ =>
                log.warn(s"closing $closeable")
            }

            closeable.close()
            it.remove()
          }
        }

        // source of sun.misc.URLClassPath is not typically distributed with the JDK
        // https://github.com/awh/openjdk7/blob/master/jdk/src/share/classes/sun/misc/URLClassPath.java
        val urlClassPathField = classOf[java.net.URLClassLoader].getDeclaredField("ucp")
        urlClassPathField.setAccessible(true)
        val urlClassPath = urlClassPathField.get(u).asInstanceOf[sun.misc.URLClassPath]

        // We need to close the Loaders (specifically the JarLoaders),
        // yet trick the URLClassPath into being re-usable. Three ways to do this:
        //
        // 1. manually close the Loaders (its a private class, tricky)
        // 2. close the loaders and then reset the collections
        // 3. close the classpath and put a new one in the classloader
        //
        // None of this is thread-safe in the slightest
        //
        // option #3 it is...
        urlClassPath.closeLoaders()
        val replacementUrlClassPath = new sun.misc.URLClassPath(u.getURLs)
        urlClassPathField.set(u, replacementUrlClassPath) // FINAL, may not be seen...

        // partial impl of option 2
        // val closedField = classOf[sun.misc.URLClassPath].getDeclaredField("closed")
        // closedField.setAccessible(true)
        // closedField.setBoolean(classPath, false)

        val parentField = classOf[java.lang.ClassLoader].getDeclaredField("parent")
        parentField.setAccessible(true)
        val parent = parentField.get(u).asInstanceOf[ClassLoader]
        if (parent != null)
          close(parent, log)

      case f: sbt.classpath.ClasspathFilter =>
        close(f.parent, log)
        close(f.root, log)

      case f: sbt.classpath.DualLoader =>
        close(f.parentA, log)
        close(f.parentB, log)

      // not accessible
      //case f: xsbt.boot.BootFilteredLoader =>

      case o =>
        log.warn(s"classloader wasn't a supported ClassLoader: ${o.getClass}")
    }
  }
}

@retronym the URLClassLoader.close() means it cannot be used again, whereas this hack puts the classloader into a reusable state. Arguably, we could call close(), then hotswap the ucp: URLClassPath instance, but that is "less" thread safe because threads could be using the ucp whereas my approach hotswaps and then cleans the old one (I say "less" because the code above is in no way thread safe).

well, unfortunately this workaround doesn't work in my work, despite it working in my standalone tests on Windows. I have no idea what is going wrong: perhaps the version of the JDK or number of cores is somehow relevant, in which case I have no idea what to do except to patch sbt to not re-use the test loader (and to close it after use).

I finally worked around this. In a nutshell, use a java agent to monkey patch URLClassLoader so that we have a much more sensible backend implementation of URLClassPath.
I think going forwards it would be very useful for both scala-compiler and scalac to have their own classloader, more similar to mine (I'm happy to discuss changing the licence if that's interesting, but we can't re-licence the OpenJDK test suite). There is a lot of optimisation can be done at the classloader level, especially when you're in a very persistent JVM process.
:microphone: drop https://github.com/fommil/class-monkey => https://ci.appveyor.com/project/fommil/sbt-big-project/build/1.0.18

I consider this closed with the above workaround


Revert "test/Services: Quarantine 'Tests with direct routing'"

This un-quarantines the direct routing K8sServices test suite, which was fixed by #18153.
Cilium-master-k8s-1.23-kernel-net-next-quarantine has not had any failures due to missing routes on Cilium nodes since the fix was merged, so the fix seems to be successful.
Note: The K8sServices test suites with across nodes are still failing with Can not connect to service "http://[ipv6]:30882" from outside cluster (1/10)) in the quarantine pipeline - but these tests remain quarantined due to cb1d3b34b75c91c9875caa90f6888cf65b03f86c according to @brb  and are not re-enabled by this PR.
This PR only re-enables tests that were failing with  Request from k8s1 to service http://[ipv6]:31872 failed, which #18153 seems to have fixed, as that particular error has not occurred anymore.
Fixes: #17895

For some reason, the bot doesn't recognize the Signed-Off-By line, it checks the wrong commit.


Fix QueueMembership bug revealed by django.test's DiscoverRunner

If HELPDESK_ENABLE_PER_QUEUE_STAFF_MEMBERSHIP was True but a user had
no QueueMembership entries, then restricting queue access generated
RelatedObjectDoesNotExist exceptions.

Ask for forgiveness whenever we try to limit a queryset by the
queuemembership related object set.

Since tests can now be run with the project's settings active,
rather than only with quicktest.py's settings, restore the initial
HELPDESK_ENABLE_PER_QUEUE_MEMBERSHIP value after having tested the
related functionality.

Closes #353; thanks for catching this @flinz!

@reduxionist wow, that was fast, good work!


Proposal for refactoring the way models are handled in Django-Shop.

Currently there are concrete and abstract models: BaseProductand Product, BaseCartand
Cart, BaseCartItem and CartItem, BaseOrderand Order and finally,
BaseOrderItem and OrderItem.
The concrete models are stored in shop.models, whereas abstract models are stored in
shop.models_bases. This is quite confusing and makes it difficult to find the right model
definition whenever one has to access the definition of one of the models.
Additionally, if someone wants to override a model, he must use a configuration directive, say
PRODUCT_MODEL, ORDER_MODEL, ORDER_MODEL_ITEM from the projects settings.py.
This makes configuration quite complicate and causes other drawbacks:

Unless someone overrides all models, the native ones appear in the administration backend
below the category Show, while the customized ones appear under the given application name.
To a customer, this backend inconsistency is quite difficult to explain.
In the past, having overriden and native models caused some issues with circular dependencies.

Therefore my proposal is to remove all the configuration settings PRODUCT_MODEL, ORDER_MODEL,
ORDER_MODEL_ITEM, etc. from django-shop. Additionally all concrete models, Product,
Order, OrderItem, Cart, CartItem are removed from django-shop. These model
definitions are just stubs, so removing them should not cause any harm. All abstract models are moved into the directory shop/models/; this is where a programmer expects them.
For an application using django-shop this should not be a big deal. Each application simply
derives each model from one of the abstract base models and thus creates its concrete model.
This also avoids another problem, one might easily encounter: Say, one starts with django-shop
and uses the built-in Cart model. In the database this is handled by a table named shop_cart.
Sometimes later, he decides to add an extra field to the model Cart. He then has to create
a customized cart model in his own application, but now the database table will have a different
name and it is not possible to override this using app_name in the model's Meta class.
These kinds of migration are not handled by South and therefore he has to write a migration for
renaming the table by hand.
Mapping of Foreign Keys
One might argue, that this can't work, since foreign keys must refer to a real model, not to
abstract ones! Therefore one can not add a field ForeignKey, OneToOneField or
ManyToManyField which refers an abstract model in the Django-Shop project. However, these
relations are fundamental for a properly working software. Imagine a CartItem without a foreign
relation to Cart.
Fortunately there is a neat trick to solve this problem. By deferring the mapping onto a real model,
instead of using a real ForeignKey, one can use a special placeholder field defining a relation
with an abstract model. Now, whenever the models are “materialized”, then these abstract relations
are converted into real foreign keys. The only drawback for this solution is, that one may derive
from an abstract model only once, but that's a non-issue and doesn't differ from the current
situation, where one for instance, also can override Cart only once.
I therefore would strongly encourage to refactor Django-Shop, since this will hugely simplify
the current model system.
Technical Details
Instead of using models.ForeignKey, models.OneToOneField or models.ManyToManyField, I
use special pseudo-fields: DeferredForeignKey, DeferredOneToOneField and
DeferredManyToManyField. During model instantiation, these pseudo-fields are replaced against
real foreign keys.
Have a look at this meta class:
https://github.com/jrief/django-shop/blob/refactor-models/shop/deferred.py
Now an abstract model can be defined as:
https://github.com/jrief/django-shop/blob/refactor-models/shop/models/abstract.py
and the materialized model then is just as simple as:
from shop.models import abstract

class RealModel1(abstract.AbstractModel1):
    pass

class RealModel2(abstract.AbstractModel2):
    pass

class RealModel3(abstract.AbstractModel3):
    pass

Gone are the days with dozens of configurations settings to hard wire the real models in Django-Shop.

I'm trying to understand the difference between django-polymorphic and DeferredForeignKey?
It was proposed to refactor the code using DeferredForeignKey, DeferredOneToOneField etc..., but from what I see this is currently using django-polymorphic
It seems conceptually similar to me, and I'd be happy to understand it a bit better...
Thanks.

Currently, https://github.com/jrief/django-shop is the release I'm working on.
Just for clarity, DeferredForeignKey and django-polymorphic are completely different. A polymorphic product has one table for its base product and additional tables for the implemented product specialization.
On the other side, the deferred pattern does nothing else, than postponing the foreign key mapping until the materialized model is known. This avoids some nasty issues, we had with circular dependencies and it allows to move the whole model tree from django-shop into your shop implementation.

Thanks for your fast answer.
From my understanding, the Deferred model doesn't allow the same polymorphic behavior, or does it?
'''
class Comment(models.Model):
commenter = models.ForeignKey('auth.User')
content = DeferredForeignKey()
body = models.TextField()
class BlogPost(models.Model):
title = models.CharField(max_length=100)
slug = models.SlugField()
body = models.TextField()
point(Comment, 'content', BlogPost)'''
It looks like from now on, all the content fields inside Comment will be of type BlogPost.
i.e: We cannot assign dynamically for each instance of Comment. It's either they are all of type BlogPost, or all of type Image (for example)
Thanks a lot for your help.

From my understanding, the Deferred model doesn't allow the same polymorphic behavior, or does it?

no, not at all
The deferred patterns only make sense for abstract models.
DeferredForeignKey  requires a foreign model.
A class using deferred keys must inherit like this
class Comment(with_metaclass(deferred.ForeignKeyBuilder, models.Model)):
    content = deferred.ForeignKey(BlogPost)
    otherfield = ...

    class Meta:
        abstract = True

Thanks a lot for your help, it's clearer now.

Disclaimer: I've just started to use django-shop and thus I'm not quite familiar with the code yet.
I've just run into an issue trying to introduce django migrations using django-shop with django 1.7 and a custom Address model (unsurprisingly, Django creates a migration in the shop app itself for whatever Address model is currently being used; I created the migration with the default Address model in place and then tried to swap in my own, which threw a reverse accessor clash for the User model).
I started digging and found that most of the django-shop models can be swapped by setting SHOP_ADDRESS_MODEL (using Address as an example). This already sounds a bit like django swappable models, which atm is used for the Auth model, but intentionally undocumented (here's why: https://code.djangoproject.com/ticket/19103; see also http://stackoverflow.com/questions/22025476/what-is-swappable-in-model-meta-for, and https://github.com/wq/django-swappable-models for a light wrapper where also the concept is explained).
If the latter is to be trusted, swappable models are already quite safe to use and I assume they'll be documented some time soon. Honestly I haven't tried, but I don't know how @jrief's solution would play with (django) migrations, which I consider important for people who want to use the default models. Also, although I bet it's a brilliant solution, I'd rather use django-builtins than reinventing the wheel with deferred fields.
As mentioned, I haven't gone into detail yet, I just have the feeling that we are moving towards a dead end (or at least a non-djangoish end). What are your thoughts on it? I'm not a code guru, but I'd be glad to help refactoring django-shop to using swappable models and django migrations (I'm about to start some experiments on that).

@febsn  Thanks for the hint. Last year I was not aware of it.
Anyway, the deferred models work quite well with migrations. The only caveat it that migrations then have to be done inside the application, but since these are models defined by the customer of that library, I can live with it.
I need a deeper look at swappable models. Currently my version of django-shop is almost ready for being released.

@jrief no wonder you weren't, it's still a bit of a mystery.
Well of course migrations can be implemented at a per-project basis, but that makes it impossible to release more complicated migrations along with the code… and as a django-shop is also suitable for smaller use cases, default out-of-the-box usable models would be an advantage, I guess.
I only realized you are member of the core team after I had written my thoughts above. Are you planning to bring your refactored version into trunk, or what exactly is the long-term plan? If there's a way for me to contribute to this issue without abandoning all your good work, let me know.

That would be great! Can we schedule a Google Hangout, then I can present some features.
What is missing? Mainly docs and tests.

Closed. Please check version 0.9.


Web is described as Experimental on pub.dev

Describe the bug
I am a new user of this library. It seems that the web platform has been stable since version 2.9.0, but there is still this statement on pub.dev: "Keep in mind that web support is still experimental".
Perhaps you should consider removing this statement?

Good catch, thanks! I've removed the warning in 6c4654754b51fdb7444b39d4687c4dd5792f4b8d, so it will disappear in the next drift version.


fix concurrent course link

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 57.764%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
279

Relevant Lines:
483

💛  - Coveralls


User did not understand what GOV.UK Verify is or what it means to sign up with it

Quote “why am I going to the GOV.UK website, I wanted to apply with Buckinghamshire”
Quote “no idea what these companies are”
Quote “I am not happy about it going to another place”
Quote “why is the Council asking someone else to verify me?”
Quote “of course I am over 20 it is an old person badge”

This is linked to issue #400. Users are unsure of what Verify does.

Thanks @colin2678 - we'll keep an issue in the CT section as well :)

#426

#465

Also linked to #462

@irinapencheva Screenshot I've just added to #461 may also resolve almost all of the user quote issues... it's already been changed in the prototype. The only one it wouldn't is “of course I am over 20 it is an old person badge” which relates to Verify hub pages. I'm not directly involved with that but am feeding local issues with Verify hub through to @anthonylord regularly. The developer I met yesterday in the Privacy training, Kevin, said the hub content is being contextualised for user relevancy but I'm not sure if that would be to the degree of adjusting security questions dependent on local service entry point or not. Definitely worth considering from a local services perspective.

@lizziebruce - yep, sounds great. Thanks for feeding into main Verify issues as well.
On this specific issue there is probably something about how the start page works as well and what's presented there about Verify  - at the moment almost nothing. This is linked to #303

@irinapencheva I will also add a line for the Start page to flag up that the Verify process could be a part of the application and identification docs may be needed... important to attempt to keep [Verify] separate from [Application] in people's minds though, in order to address the 'why do I need all this for a bus pass?' issue which came up a lot in UR. In actual fact they don't acutally need to answer the security questions, but they have chosen to in order not to scan in documents. I think if we can make that choice more apparent people will be less frustrated/baffled, as they made the choice to go through Verify.

@lizziebruce - yep, fitting in with clearly creating that mental picture in people's mind of what will happen! And on service start page might be worth saying what GOV.UK Verify is. I've seen it other services as 'the new way to prove who you are online' or sth along those lines. Ant will have a good insight on that

@irinapencheva "prove who you are online" is good. I'm not sure we can say "the new way" anymore. Ant described Verify to me as 'a kind of online passport' which really clicked for me but unfortunately is too metaphorical to use in service copy.

@lizziebruce - agreed.. not new anymore, so happy with any wording you'd think would work

@irinapencheva
Content redesign for Start page. Complements content redesign of pre-Verify page to embed the idea of needing to choose either Verify or docs scan from the very first page. Also flags need to upload a photo. This content has been moved above the Apply button to increase the liklihood of it getting noticed.

@lizziebruce - on the 1st point of 'About your application' do you think the issue about eligibility should also be mentioned?
Something along the lines of: "Identify yourself and prove your eligibility using ..."

@irinapencheva is there a user need for adding "prove your eligibility"? Has anyone been asking why they have to identify themselves? If so "identify yourself and prove your eligibility" sounds good.

@lizziebruce - overall people don't seem to understand why they have to identify themselves and feel they're entitled to the pass (as seen in last round of UR), so thinking it might be worth testing overall comprehension of why they're going through Verify if the connection between proving identity and eligibility is more explicit

@irinapencheva sure, that makes sense. I edit the copy to reflect that. Also needs further adjusting as it is the first mention of Verify and there's no explanation of what Verify is. It may work better to just say answer security questions and not mention Verify here. (I showed people on the GDS design training this page and got some feedback re no explanation of Verify here too and it came up in UR I believe.)

I think taking the mention of Verify out of the start page could be a good idea because we don't have the space to properly explain it there.
Taking this back to the original issue of making sure users understand why Verify is being used, I agree with Lizzie that the changes made in #395 since this issue was created should hopefully solve the problems reported, so let's move this into "Recent changes have been made" until we see this confusion about Verify come up again.

@lizziebruce @sanjaypoyzer Just wanted to highlight a couple of things I've seen in the past that might be relevant.

In the past we've tried to avoid saying Verify is to let users prove who they are, as the inferred assumption is then that we don't actually trust our users. Instead we've tried to say things like 'Verify stops someone pretending to be you'
The other thing was about using the phrase 'the new way to...' When I've seen this tested before, it's gone some way to addressing people's concerns about Verify working in unexpected ways. It's new, so it's ok for it to be a bit unconventional

@irinapencheva discussing sounds good!

@irinapencheva @petegale
Sorry I thought I'd progressed this further. Suggest the "About" copy is refined further to:

About your application

Apply by answering security questions or scanning documents.
Upload a digital photo – we'll give you guidance on doing this.
Submit your application online.
We'll send your bus pass by post.

We spoke about perhaps moving this onto its own page to help users focus on reading it aka not skip it. What do you think @sanjaypoyzer @petegale?

@lizziebruce Hi Lizzie, just a thought, but could we try removing the word digital and just say upload a photo. The word digital seemed to cause confusion?

@TracyEaton Hi Tracy, that issue is more related to #301 - I'll respond to your question there.

@TracyEaton Hi Tracy, thanks. I think later in the journey users asked why they weren't told up front about what they would need, and so if we just put "photo" they might think they're all set when they're not.
The photo upload users-felt-discouraged-from-start has its own issue #458 and the guidance help flow is in 301, as Sanjay's mentioned too.

These changes to the prototype (screenshot below) should help the original issue "Users need to use GOV.UK Verify but did not understand what GOV.UK Verify is or what it means to sign up with it", but some of the UR comments relate directly to the Verify hub pages so have added that label too.
Before

After

Are we able to localise these screens? Our existing CT process does not usually require a utility bill,passport or driving licence and takes considerably less time than 20 minutes.

@colin2678 could you tell us what documentation do you do require and how long your CT process is?

We require a name, address, dob and a NI number to make an experian check. along with a photo upload or to take a photo using their webcam. This process usually takes around 4 minutes.

@colin2678 Hi Colin, just to clarify, are you talking about the alternative to Verify, ie the "Scan and upload documents" route?

If the check fails then customers must send in proof of identity document (a passport, driving license, birth certificate in the same name as your application) and a proof of address  (a utility bill, bank statement, or any other official letter).

@sanjaypoyzer Yes this is our existing product: https://concessioncard.net/Register_input.action It has it's flaws, but speed is not one of them.

@colin2678 Hi Colin, I think yes that would need to be a localisation as most councils require document upload which can take users longer. We have out "up to" in front of 20 minutes to account for speedier / more digitally savvy users.

@petegale - Thanks for closing this issue. I will reopen however #422 as the evidence for this being addressed is still inconclusive

@petegale As discussed on the call I recognise that there was better take-up of the Verify option, but did not see evidence that most CT users understood what Verify would do. To me this is a fundamental issue that creates confusion in CT users later on in the Verify process. I do not believe we can close this issue on the back of on 2 UR sessions.

@colin2678 - We are happy to reopen this and test further at the next session

See front-loading of explanation about companies in #588 #602


markdown cells should flag spelling errors

this is one of the nice things about text cells now, when writing prose it's nice to get the red underline indicating a spelling error.

Looks like this is never going to happen so long as we use codemirror for markdown
https://github.com/codemirror/CodeMirror/issues/1017#issuecomment-52287597

How about https://github.com/cfinke/Typo.js?


Corrected version of terraform-plugin-sdk replace in provider/go.mod

Fixed issue where a previous PR introduced an incorrect version in provider/go.mod.

No problems @stack72 ! :) thanks for merging so quickly.


Disable drag to dismiss

Hi @kevin0571 thanks for the library...how is it possible to disable drag to dismiss option...

@hardikamal
You can disable drag to dismiss with: self.popupController.navigationBar.draggable = NO.

Thanks it worked...


Make ReactCSS either a high order component or a decorator

Or maybe both?

A couple things I've seen using react-css in production:
classes method gets huge, quick. this could be a side effect of having too large of classes in the first place, but what i like about inline styles is that the styles are defined with the component. what i found was that we ended up with these huge classes functions defining all of these styles and they became unwieldy fast.
i think class components themselves tend to become really large. i've started using stateless components which make composing components much easier. i'm not sure how these work with the current API of ReactCSS (i know v2 was on the horizon at some point).
i haven't looked into the technical details, but something like radium's api: https://github.com/FormidableLabs/radium#usage is interesting because i could do something like:
const MyStatelessComponent = () => {
      const styles = {text: {color: 'black'}};
      return
"hi"
;
}

export { MyStatelessComponent };
export Radium(MyStatelessComponent);

if ReactCSS is a HoC we could do the same thing which would definitely be interesting.

While switching over to ReactCss for my personal website, I created a basic higher-order component, see https://gist.github.com/nheyn/ac4f4200313fd34d81ee.
It is a little bit of a hack, I had to make a "dummy Component" to bind to the inline(...) function. It was the only way to get the classes object from the Component (that was passed to the higher-order component) and return it from this.classes(...) in inline(...).
Other then that, it's similar to Component.js. The main differences are:

The css(...) method is added to the props of the passed in Component
The css(...) method (now a prop) must be passed the classes object (I added it as the 1st argument)
The styles(...) method must always be defined

Here is one of the Components (with some simplifications) that uses the higher-order component:
import React from 'react';
import useStyles from '../useStyles';

const Section = React.createClass({
  contextTypes: {
    screenSize: React.PropTypes.string.isRequired
  },
  styles() {
    const { screenSize } = this.context;

    return this.props.css(this.classes(), {
      smallScreen: screenSize === 'xs' || screenSize === 'sm'
    });
  },
  classes() {
    return {
      'default': {
        section: {
          position: 'absolute',
          left: '10%',
          right: '10%',
          heynioColors: 'default'
        }
      },
      'smallScreen': {
        section: {
          left: 0,
          right: 0
        }
      }
    };
  },
  render() {
    const style = this.styles();

    return (

        {this.props.children}

    );
  }
});

export default useStyles(Section);

The API has been changed in 1.0.0 to make this unnecessary!


Move recrusion Program into risc0-binfmt

This PR moves the Program struct, describing the predicates which act as the executable that runs on the recursion machine circuit, into the risc0-binfmt crate. This supports bootstrapping the circuit as it allows the code in the circuit repository to import the Program struct without importing risc0-zkvm, which would introduce a circular dependency.

I am going to rewrite this PR to move the control ID calculation for the recursion predicates from WIP bootstrap to xtask bootstrap.

thanks for info

I am going to rewrite this PR to move the control ID calculation for the recursion predicates from WIP bootstrap to xtask bootstrap.

Finished this refactor. This PR is fully ready for review.


feature: cast features to float (ArangoDB to PyG)

(creating a dead issue for documentation purposes, as this was ruled out via internal conversation)
When going from ArangoDB to PyG, a cast_features_to_float parameter can be passed to indicate which tensors should be casted to float (e.g torch.tensor([1]).float()).
For example:
data = FakeHeteroDataset(edge_dim=2)[0]
adbpyg_adapter.arangodb_to_pyg('PyG Graph', data, cast_features_to_float=['x', 'edge_attr'])

Assuming the ArangoDB data corresponding to the 'x' and 'edge_attr' feature matrices were not of type float, providing cast_features_to_float would indicate to the adapter to cast the created tensors to float.

As per our internal conversations with ArangoML members, this has been ruled out.
Reason: It was argued that given the supported data types in torch, cast_features_to_float would open a slippery slope to cast_features_to_complex, cast_features_to_integer, cast_features_to_bool, etc..


Print Feature Crashes Browser

Each time I try and print my browser crashes and I have to restart the browser

Please submit a bug report.


The check for max column in a line not calculate the unicode correctly.

If the code contains unicode character, in editor the column if far less than the max column, bug the luacheck will report error.

Part of #45 again. Sorry, I couldn't work on this yet.


Config file should come from default branch

The config file should be pulled from the default branch or the target of the pr. Right now it comes from the current branch where the PR is created from

The configuration for the action is now stored in the workflow yaml. That comes from the default branch.
This issue will be relevant if branch specific configuration is added again in the future.
Closing for now.


proposal: crypto/tls: support TLS 1.3 post-handshake authentication

What version of Go are you using (go version)?

$ go version
go version go1.14.5 darwin/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/felix/Library/Caches/go-build"
GOENV="/Users/felix/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/felix/workspace/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14.5/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/felix/workspace/tests/go-tls-pha/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/gr/j517787d56s73w50gnpv08gr0000gn/T/go-build528430427=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
I use the Go TLS client to implement a client for an endpoint that requires authentication with client certificates. The endpoint decides if client authentication is needed based on the path in the HTTP request. Therefore, the server can only decide if client authentication is needed after the TLS connection has been fully set up.
With TLS 1.3, the server would achieve the authentication with client certificates using post-handshake authentication. However, the Go TLS client does not support post-handshake authentication.
For reproduction of the issue, one can use the Apache httpd (I tested with version 2.4) with the config including the snippet below and a small go program like the one below.
httpd.conf snippet
<VirtualHost *:443>
    LogLevel debug

    SSLEngine on
    SSLCertificateFile /usr/local/apache2/conf/server.crt
    SSLCertificateKeyFile /usr/local/apache2/conf/server.key

    SSLVerifyClient none
    SSLCACertificateFile /usr/local/apache2/conf/ca.crt
    <Location "/test-cert.html">
        SSLVerifyClient require
        SSLVerifyDepth 1
    </Location>
</VirtualHost>

client.go
package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"os"
	"time"

	"go.etcd.io/etcd/pkg/transport"
)

func main() {
	f, err := os.Create("./tls.keylog")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()

	tlsInfo := transport.TLSInfo{
		TrustedCAFile: "./ca.crt",
		CertFile:      "./client.crt",
		KeyFile:       "./client.key",
	}
	tlsTransport, err := transport.NewTransport(tlsInfo, 30*time.Second)
	if err != nil {
		log.Fatal(err)
	}
	tlsTransport.TLSClientConfig.KeyLogWriter = f

	client := &http.Client{
		Transport: tlsTransport,
	}

	r, err := client.Get("https://localhost/test-cert.html")
	if err != nil {
		log.Fatal(err)
	}

	defer r.Body.Close()
	body, err := ioutil.ReadAll(r.Body)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%s\n", body)
}

What did you expect to see?
I would have expected the TLS client to support TLS 1.3 post-handshake authentication and be able to connect to the endpoint.
What did you see instead?
The server closes the connection to the client as soon as it encounters that a client certificate would be required for a specific route. In the httpd logs this error can be found:
[ssl:debug] [pid 7:tid<PHONE_NUMBER>97632] ssl_engine_kernel.c(1148): [client <IP_ADDRESS>:35172] AH10129: verify client post handshake
[ssl:error] [pid 7:tid<PHONE_NUMBER>97632] [client <IP_ADDRESS>:35172] AH10158: cannot perform post-handshake authentication
[ssl:error] [pid 7:tid<PHONE_NUMBER>97632] SSL Library Error: error:14268117:SSL routines:SSL_verify_client_post_handshake:extension not received
<IP_ADDRESS> - - [31/Jul/2020:19:40:50 +0000] "GET /test-cert.html HTTP/1.1" 403 258

This happens because the TLS client does not indicate support for post-handshake authentication in the ClientHello (which is actually correct, as it is not supported right now).
After looking at the TLS 1.3 client implementation, I think adding post-handshake authentication support for the TLS 1.3 client should be possible and I would be happy to help with this.

I tried to implement the post-handshake authentication for the TLS 1.3 client. The change above supports at least the server configuration from the issue.
The change is just a draft right now. It does not contain any tests, probably breaks tests, and blindly indicates support for post_handshake_auth. However, it is useful to see which areas are a bit difficult to deal with right now:

The code for sending Certificate, CertificateVerify and Finished needs to be called from Conn
The transcript of all handshake messages needs to be available from Conn. Right now, it is just available from clientHandshakeStateTLS13

Thank you for this question @skyfmmf and welcome to the Go project!
Kindly ping some cryptography experts @katiehockman @FiloSottile to take a look.

Thoughts, @FiloSottile ?

Ping, @FiloSottile

Ping, @FiloSottile

It would be useful to know how commonly required this is. I assume web servers do this when in TLS 1.2 they would have asked for a renegotiation?
The main complexity costs of this would be:

keeping handshake state past the end of the handshake
on the client, invoking GetClientCertificate after the end of that handshake, which might be unexpected by the application
on the server, exposing a whole new API

It's unlikely we'll implement this on the server side unless it's a very common request. On the client, we can consider it, like we did for (sigh) renegotiation. I wonder if (1) would affect every connection or if the server acknowledges interest in post-handshake auth.

It would be useful to know how commonly required this is. I assume web servers do this when in TLS 1.2 they would have asked for a renegotiation?
The main complexity costs of this would be:

keeping handshake state past the end of the handshake
on the client, invoking GetClientCertificate after the end of that handshake, which might be unexpected by the application
on the server, exposing a whole new API

It's unlikely we'll implement this on the server side unless it's a very common request. On the client, we can consider it, like we did for (sigh) renegotiation. I wonder if (1) would affect every connection or if the server acknowledges interest in post-handshake auth.

Yes, in the place where we now see a request for PHA, with TLS 1.2 the server asked for a renegotiation. Renegotiation support in the TLS 1.2 client is also opt-in in Go, if I am not wrong? So this opt-in behavior is what you could imagine for PHA in the TLS 1.3 client as well?
For the client side, at least Apache considers it a client bug not to support PHA (https://bz.apache.org/bugzilla/show_bug.cgi?id=62975). As it affects some connections to the commonly used Apache httpd, I think PHA on the client side may be a common requirement. On the server side, I do not know, how commonly required it is.
I do not recall if the server in any way indicates that it supports PHA in the handshake, but I will check it later on. At least the RFC does not sound like the server would acknowledge the indicated client support.

Yes, in the place where we now see a request for PHA, with TLS 1.2 the server asked for a renegotiation. Renegotiation support in the TLS 1.2 client is also opt-in in Go, if I am not wrong? So this opt-in behavior is what you could imagine for PHA in the TLS 1.3 client as well?
For the client side, at least Apache considers it a client bug not to support PHA (https://bz.apache.org/bugzilla/show_bug.cgi?id=62975). As it affects some connections to the commonly used Apache httpd, I think PHA on the client side may be a common requirement. On the server side, I do not know, how commonly required it is.
I do not recall if the server in any way indicates that it supports PHA in the handshake, but I will check it later on. At least the RFC does not sound like the server would acknowledge the indicated client support.

Considering it a client bug seems excessive. It is definitely not a mandatory extension. https://tools.ietf.org/html/rfc8446#section-9.2
Chrome has not implemented it and Firefox gated it behind the off-by-default security.tls.enable_post_handshake_auth setting. https://bugs.chromium.org/p/chromium/issues/detail?id=911653#c4
The main problem with it is that it is a layering violation if it's not tightly integrated with the application protocol (like in HTTP/1.1) or if the application protocol is multiplexed (like HTTP/2). Indeed, there is an RFC forbidding its use with HTTP/2. https://www.rfc-editor.org/rfc/rfc8740.html
I don't think this is something we'd want to implement, since it seems it has a complicated deployment, compatibility, and safety story. I believe there are other efforts to bring post-handshake authentication to HTTP/2, although I have not checked in on them in a while.

Considering it a client bug seems excessive. It is definitely not a mandatory extension. https://tools.ietf.org/html/rfc8446#section-9.2
Chrome has not implemented it and Firefox gated it behind the off-by-default security.tls.enable_post_handshake_auth setting. https://bugs.chromium.org/p/chromium/issues/detail?id=911653#c4
The main problem with it is that it is a layering violation if it's not tightly integrated with the application protocol (like in HTTP/1.1) or if the application protocol is multiplexed (like HTTP/2). Indeed, there is an RFC forbidding its use with HTTP/2. https://www.rfc-editor.org/rfc/rfc8740.html
I don't think this is something we'd want to implement, since it seems it has a complicated deployment, compatibility, and safety story. I believe there are other efforts to bring post-handshake authentication to HTTP/2, although I have not checked in on them in a while.

I totally agree that considering it a client bug is a bit too much, given that it is not mandatory to implement.
I was not aware that the browsers mostly do not implement PHA. However, I would argue that web browsers are not generally the best reference when it comes to TLS client certificate support. I see client certificates more in use with machine-to-machine communication over HTTPS.
As PHA is standardized and used by at least one common server implementation, I still see that it could be relevant to implement the client side. To me it appears to be less of a mess and security issue than renegotiation. But I also understand that it is not a universally needed feature that adds complexity to the implementation and may therefore not be desired. Maybe we can wait a bit to see if there are further requests to implement it?

I totally agree that considering it a client bug is a bit too much, given that it is not mandatory to implement.
I was not aware that the browsers mostly do not implement PHA. However, I would argue that web browsers are not generally the best reference when it comes to TLS client certificate support. I see client certificates more in use with machine-to-machine communication over HTTPS.
As PHA is standardized and used by at least one common server implementation, I still see that it could be relevant to implement the client side. To me it appears to be less of a mess and security issue than renegotiation. But I also understand that it is not a universally needed feature that adds complexity to the implementation and may therefore not be desired. Maybe we can wait a bit to see if there are further requests to implement it?

I agree client certificate support is probably more relevant to us than to browsers, but I'm not sure that extends to PHA. Most of the times client certificates are used in mTLS configurations where a certificate is negotiated at handshake time.
It is definitely less of a mess than renegotiation, but I still don't think it carries its own weight, due to the HTTP/2 incompatibility. Since applications already have to disable HTTP/2 if they want these semantics, they might as well also disable TLS 1.3 for now.
Happy to put this proposal on hold to collect further requests and gauge interest.

I agree client certificate support is probably more relevant to us than to browsers, but I'm not sure that extends to PHA. Most of the times client certificates are used in mTLS configurations where a certificate is negotiated at handshake time.
It is definitely less of a mess than renegotiation, but I still don't think it carries its own weight, due to the HTTP/2 incompatibility. Since applications already have to disable HTTP/2 if they want these semantics, they might as well also disable TLS 1.3 for now.
Happy to put this proposal on hold to collect further requests and gauge interest.

Placed on hold.
— rsc for the proposal review group

Placed on hold.
— rsc for the proposal review group

Ran into the same issue today. I was able to work around this, by forcing TLS 1.2 for now, but it would be great to use TLS 1.3 as well. So I'd gladly add my "+1" to the requester list.


Fix parameter order.

The order of parameters seems to be wrong right now within the Cqrs class.
The data is correct but the names are not.
This is what I get when starting the app:

Where Pricing::AddItemToBasket is a command (not a command handler):

And Pricing::OnAddItemToBasket is the command handler (not a command):

This PR contains a small fix for that both for #register_command and #register. :+1:
Let me know if I got it wrong. Thanks!

Well spotted, thanks!

No worries!
Really appreciate the sample by the way.
Cheers.


Display the backtrace of errors to help debug them

Updated to use a variable GOSPELDEBUG so that setting OCAMLRUNPARAM=b does not automatically trigger that behaviour in Gospel.


profile picture url accepting links that aren't picture formats

Reproduce:
Register User, ie John Doe
Change profile pic url from .png to .com
url is still accepted

patched bug by adding checks into user service and user schemas


[pwm, rtl] RTL update for issue #6044 pare down PWM CDC

RTL update to pare down PWM CDC

Remove PWM CDC module, use TLUL async fifo for clock domain crossing
Move reg_top and everything else to core clock domain
Delete file pwm_cdc.sv
Fixes #6044

Signed-off-by: Muqing Liu<EMAIL_ADDRESS>

@Muqing-Liu there some lint issues, run "'util/dvsim/dvsim.py -t veriblelint hw/top_earlgrey/lint/top_earlgrey_lint_cfgs.hjson" to get the error and fix.

Thanks for the PR @Muqing-Liu! This LGTM.
Please let me know when you are ready to finalize and I can merge it for you.

Oh actually it looks like there are conflicts. Can you please rebase this (and fix any conflicts)?

Thanks for the PR @Muqing-Liu! This LGTM.
Please let me know when you are ready to finalize and I can merge it for you.

Oh actually it looks like there are conflicts. Can you please rebase this (and fix any conflicts)?

Sure, I have rebased it and made some minor changes. I have pushed the updates, please take a look. Thanks!

@Muqing-Liu @martin-lueker we have a general cdc proposal circling around for the regfile.
do you guys want to have a look and see what you think? It might save you some trouble here.

@Muqing-Liu @martin-lueker we have a general cdc proposal circling around for the regfile.
do you guys want to have a look and see what you think? It might save you some trouble here.

By "save trouble" do you mean: "The tlul_fifo approach here is about to be deprecated, so please don't bother?"

@Muqing-Liu @martin-lueker we have a general cdc proposal circling around for the regfile.
do you guys want to have a look and see what you think? It might save you some trouble here.

By "save trouble" do you mean: "The tlul_fifo approach here is about to be deprecated, so please don't bother?"

This is AFAICT done for the tlul_fifo approach, so unless the this approach is soon to be unsuitable/non-standard, I'd personally prefer to see this merged and call this issue closed.

@Muqing-Liu @martin-lueker we have a general cdc proposal circling around for the regfile.
do you guys want to have a look and see what you think? It might save you some trouble here.

By "save trouble" do you mean: "The tlul_fifo approach here is about to be deprecated, so please don't bother?"

This is AFAICT done for the tlul_fifo approach, so unless the this approach is soon to be unsuitable/non-standard, I'd personally prefer to see this merged and call this issue closed.

we talked it over with the software team, and basically for clocks this slow (in the KHz range) we cannot use the tlul approach.
The potential latency is just too long (probably 20~30us per write/read), EVEN if it's only for setup.  So the new cdc proposal is built directly into the regfile. Hopefully the user just specifies which registers they want async, and everything else will be handled.
Don't close this PR yet, @eunchan will go over the proposal on Monday. If we have general agreement this is the way to go, then we can switch over.  If not, then we may have to revisit this PR and figure out how to deal with the latency issues.

Hi @Muqing-Liu,
After hearing a bit more from @tjaychen, about the issues they've identified, this tlul-fifo approach has some major drawbacks, both in PWM and in the aon-timer.   The issue being that if the IP core clock is very slow the TLUL latencies will be unacceptably large.  @tjaychen will be presenting more on this at the next Silicon meeting.
I think some of the work here will be useful, but the CDC will be specified in the HJSON, not explicitly done in the IP.

I have also simulated the pwm module and it can work correctly with the new CDC. But during the debugging, I found there seems to be some issues with the generated pwm_reg_top.sv file. As shown in the figure below, the name of the sync_core_update was wrong, the letter "c" was omitted, and the issue was the same for all prim_subreg_async modules in this pwm_reg_top.sv file, so I have to manually update the parameter name.

I have also simulated the pwm module and it can work correctly with the new CDC. But during the debugging, I found there seems to be some issues with the generated pwm_reg_top.sv file. As shown in the figure below, the name of the sync_core_update was spelt wrong, the letter "c" was omitted, and the issue was the same for all prim_subreg_async modules in this pwm_reg_top.sv file, so I have to manually update the parameter name.

@Muqing-Liu that was my mistake, i didn't strip some of the prefix correctly.
A fix for that (and another fixes) is currently happening under #7850 , stay tuned!

I have also simulated the pwm module and it can work correctly with the new CDC. But during the debugging, I found there seems to be some issues with the generated pwm_reg_top.sv file. As shown in the figure below, the name of the sync_core_update was spelt wrong, the letter "c" was omitted, and the issue was the same for all prim_subreg_async modules in this pwm_reg_top.sv file, so I have to manually update the parameter name.

@Muqing-Liu that was my mistake, i didn't strip some of the prefix correctly.
A fix for that (and another fixes) is currently happening under #7850 , stay tuned!

Great! Thank you very much for letting me know.

@Muqing-Liu can you rebase one more time now? the problem you were showing earlier should be gone.

@Muqing-Liu can you rebase one more time now? the problem you were showing earlier should be gone.

Yes, sure, I have rebased it and updated the pwm_reg_top file. Thanks a lot for your updates!

@Muqing-Liu: Thanks for taking care of those CI issues.

Everyone else: Are there any further comments or requests for this PR?


⚠ Bofrost.ch has degraded performance

In 00eac23, Bofrost.ch (https://www.bofrost.ch) experienced degraded performance:

HTTP code: 200
Response time: 1181 ms

Resolved: Bofrost.ch performance has improved in 0223c5e.


Feature Request: Support building shared/static libraries

Added support for building as shared/static libraries
Export symbols for Windows dll

thank you for your pr, i appov it to main ( cf91f2b0937b5bd02178b09bfc660f2022fffc43 )
in order you can use this version in CGraph.
and write your name to contributors: https://github.com/ChunelFeng/CGraph/blob/126f61167d4f6f6552bac6af1736e1c718534704/CONTRIBUTORS.md
but actually we do not need this, we really recommend user use CGraph by 3rd-party with source code.
thank you all the same. please let me know where are you from if convenient.

Thanks, will add as 3rdparty

please let me know in which case or project, you consider to use CGraph,
maybe i can give you some help


Request: Include location in naming

Can you include the location as a 3 or 4 letter abbreviation? It has to be check how many letters are required to make all locations unique.
Expected behavior
Add an option to module configuration location, e.g.
module "naming" {
  source = "Azure/naming/azurerm"
#...
  prefix = "hello"
  location = "westus2"
#...
}

So the generated module.naming.resource_group.name is "hello-rg-wu2" (format "[prefix]-[resource-type]-[location]-[suffix]-[unique]"). For global resources to be consistent I suggest to use an abbreviation as well, or leave it empty. Maybe the community has an opinion on that.

@fsettele I usually include the location in the suffix list when it makes sense to do so, and that's worked well in my experience.
It gives you the flexibility to use a short location name like wu2 or whatever naming convention your organization agrees upon.
module "naming" {
  source = "Azure/naming/azurerm"
  suffix = [var.short_location, var.environment, var.app_name]
}

module.naming.resource_group.name => rg-wu2-prod-navigator

I think @travisty- Got it correct here no need to add this as a specific argument.


CPU / RAM issue

Hello,
I have a massive issue here with Microsoft Teams.
I have a i7 8700k 4.8Ghz and 32Go of RAM.
I don't know more about the issue. Microsoft Teams was idle in the background while I was playing a game. And it crashed my PC in a second.
I tried to take a screenshot of it, but it was full lagging so I don't have everything on the screenshot, because it was taking 32Go of RAM and not 12. Also the CPU was waving between 80 and 100%.

As you can see there is multiple Microsoft Teams, but I launched only a single instance.
It takes an average of 10 hours for it to get crazy, it's random. But when it starts to get crazy, it takes 100% resources in a second.
I stay available if you need more informations.
Best regards,
Vincent

@Farcry69 Thanks for reaching out to us. We are looking into this.

@Farcry69 Please use the report a problem feature under "Help" to report this issue.


Forcing usage of a specific codec

Have you read the Tutorials?
Yes
Have you read the FAQ and checked for duplicate open issues?
Yes
What version of Shaka Player are you using?
2.5.13
Please ask your question
I'm transcoding both H.264 and H.265 in the same manifest. On a device that has proper support for H.265 Shaka Player is filtering out the H.265 variants and only using H.264. Is there a way to force the use of a specific codec (H.265 in this case) if there is support?
My specific use case is in a Vizio TV app that can only stream higher than 1080p if the H.265 codec is used, however, both H.264 and H.265 variants are needed for wide support of 4k and non-4k TVs. This was previously working when I was using a higher H.264 Profile Level but lowered some variants for better TV compatibility and now shaka player is always choosing H.264.

There isn't an "official" way to do it, although we have an open feature request for this: #2179
I might be able to suggest a workaround to accomplish this though.
We use the initial stream choice to lock in the selected codec. If you manually select a track with the codec you want through our selectVariantTracks() API before we got a chance to pick anything, your selected codec will be the one locked in! :)
You can listen to our streaming event to select the initial track before we started streaming.
Call player.getVariantTracks() to get all the tracks, then select any one of them that has the videoCodec you want, and call player.selectVariantTrack() with it.
Does this make sense?
#2179 isn't scheduled yet, but I think it might get worked on in Q4

Makes perfect sense. Thanks for the workaround.

@ismena Testing this workaround and it appears that the variant tracks are already filtered when the streaming event is fired. I'm only seeing tracks for a single (h264) codec.

@mikemerritt Ah, sorry. Try the 'manifestparsed' event instead!

Hmm. That was my best idea :(
Let's get the big bros, see if they have more thoughts :)
@joeyparrish @TheModMaker any thoughts on what else might be a good stopping point to force a variant before the initial stream set is selected?

Sorry, there does not seem to be a way to force the selection of a codec through the Shaka Player API at this time.  #2179 will track the feature request for that.
One (ugly) way you could hack it to do what you want would be to patch MediaSource.isTypeSupported.  For example, to disable AAC, you could do this:
const origIsTypeSupported = MediaSource.isTypeSupported;
MediaSource.isTypeSupported = (type) => {
  if (/codecs="mp4a.40.2"/.exec(type)) {
    return false;
  }
  return origIsTypeSupported.call(MediaSource, type);
};

That will then affect all filtering operations done by Shaka Player.  It's not ideal, but we don't have a mechanism for this yet.

@joeyparrish This works, thanks!.
For anyone else in this situation, here is what I ended up doing.
      const origIsTypeSupported = MediaSource.isTypeSupported;
      const isHvcSupported = origIsTypeSupported.call(MediaSource, 'video/mp4;codecs="hvc1.1.6.L153.B0"');

      MediaSource.isTypeSupported = (type) => {
        if (isHvcSupported && /codecs="avc/.exec(type)) {
          return false;
        }
        return origIsTypeSupported.call(MediaSource, type);
      };

I check for HVC (H.265) support. Then patch MediaSource.isTypeSupported with a function that filters out all avc (H.264) types if HVC is supported.


torvalds.xyz

This repo is a mirror and does not accept direct pull requests. Please send contributions formatted with git format-patch to<EMAIL_ADDRESS>This particular change is small enough I committed it myself.


Fcitx 5 doesn't work on Wayland

Describe the bug
Fcitx 5 doesn't work on Wayland.
To Reproduce
Enable GNOME 4 & Fcitx 5.
{
  services.xserver = {
    displayManager.gdm.enable = true;
    desktopManager.gnome.enable = true;
  };
  i18n.inputMethod = {
    enabled = "fcitx5";
   fcitx5.addons = with pkgs; [ fcitx5-mozc ];
  };
}

Configure Fcitx with Mozc
Make sure no browser processes (ex. google-chrome, firefox) are not running
Open a browser app via the GNOME application launcher
Can't enable Mozc in the browser session (can't switch input method, can't make Mozc working even if we switch the input method by fcitx5-remote)
Kill the browser sessions
Open a browser app via command line in a terminal
This time we can make Mozc working

Expected behavior
Fcitx 5 & Mozc is working in a browser session run by the GNOME application launcher.
Notify maintainers
@poscat0x04 @Vonfry
Metadata
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.45, NixOS, 21.11.20210626.f84d5fb (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 3.0pre20200829_f156513`
 - channels(root): `""`
 - channels(sei40kr): `"nixpkgs-21.05.1205.b72bde7c4a1"`
 - nixpkgs: `/home/sei40kr/.nix-defexpr/channels/nixpkgs`

Did you re-log in after enabling fcitx? It sets some environment variables and GNOME Shell will not see them until restarted, whereas terminal will source profile on startup, getting the latest enviromnent.

Also is this Wayland or X11 GNOME?

As talking above, I think the problem is that when gnome starts, fcitx5
isn't started or the environment variable isn't passed, therefore
applications don't handle special action for input method.
You can open a pure terminal in gnome to have environment run in gnome,
for example, not loading any profiles and initial scripts like .bash_profile,
.bashrc and /etc/profile. Then run fcitx5-diagnose. Note, runing bash
--noprofile with a opened temrminal is useless, because the environment
variable in profiles are inherited.
By the way, I use xmonad without any desktop managers, and fcitx5 is
worked fine. So does my friend who uses KDE.
Seong Yong-ju @.***> writes:

Describe the bug
Fcitx 5 don't work in browser apps run by the GNOME application launcher.
To Reproduce
Enable GNOME 4 & Fcitx 5.
{
  services.xserver = {
    displayManager.gdm.enable = true;
    desktopManager.gnome.enable = true;
  };
  i18n.inputMethod = {
    enabled = "fcitx5";
   fcitx5.addons = with pkgs; [ fcitx5-mozc ];
  };
}

Configure Fcitx with Mozc
Make sure no browser processes (ex. google-chrome, firefox) are not running
Open a browser app via the GNOME application launcher
Can't enable Mozc in the browser session (can't switch input method, can't make Mozc working even if we switch the input method by fcitx5-remote)
Kill the browser sessions
Open a browser app via command line in a terminal
This time we can make Mozc working

Expected behavior
Fcitx 5 & Mozc is working in a browser session run by the GNOME application launcher.
Notify maintainers
@poscat0x04 @Vonfry
Metadata
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.45, NixOS, 21.11.20210626.f84d5fb (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 3.0pre20200829_f156513`
 - channels(root): `""`
 - channels(sei40kr): `"nixpkgs-21.05.1205.b72bde7c4a1"`
 - nixpkgs: `/home/sei40kr/.nix-defexpr/channels/nixpkgs`

Thank you for your quick responses.
@jtojnar

Did you re-log in after enabling fcitx?

Yes, I tried.
@jansol

Also is this Wayland or X11 GNOME?

Wayland
@Vonfry
I tried to press Alt+F2 and open sh, and run env | less there.

Results of env | less

LD_LIBRARY_PATH=
XDG_MENU_PREFIX=gnome-
LANG=en_US.UTF-8
GDM_LANG=en_US.UTF-8
MANAGERPID=2319
DISPLAY=:0
INVOCATION_ID=ba8e976c7a5e4606bc76cc8a239bd26a
ALACRITTY_LOG=/tmp/Alacritty-321428.log
LOCALE_ARCHIVE_2_27=/nix/store/4w3l4vbgm0wj04mz38ifxh14s7s7qc5y-glibc-locales-2.32-48/lib/locale/locale-archive
COLORTERM=truecolor
USERNAME=sei40kr
MOZ_PLUGIN_PATH=/home/sei40kr/.nix-profile/lib/mozilla/plugins:/etc/profiles/per-user/sei40kr/lib/mozilla/plugins:/nix/var/nix/profiles/default/lib/mozilla/plugins:/run/current-system/sw/lib/mozilla/plugins
KDEDIRS=/home/sei40kr/.nix-profile:/etc/profiles/per-user/sei40kr:/nix/var/nix/profiles/default:/run/current-system/sw
NIX_GSETTINGS_OVERRIDES_DIR=/nix/store/pwpy2jg8kgb12nlkfqp4x3vndfar7lw6-nixos-gsettings-desktop-schemas/share/gsettings-schemas/nixos-gsettings-overrides/glib-2.0/schemas
QTWEBKIT_PLUGIN_PATH=/home/sei40kr/.nix-profile/lib/mozilla/plugins/:/etc/profiles/per-user/sei40kr/lib/mozilla/plugins/:/nix/var/nix/profiles/default/lib/mozilla/plugins/:/run/current-system/sw/lib/mozilla/plugins/
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
USER=sei40kr
DESKTOP_SESSION=gnome
TZDIR=/etc/zoneinfo
CUPS_DATADIR=/nix/store/jq8z5l622ib2ciqp6ymnlz8qw7m25xcs-cups-progs/share/cups
LIBEXEC_PATH=/home/sei40kr/.nix-profile/lib/libexec:/etc/profiles/per-user/sei40kr/lib/libexec:/nix/var/nix/profiles/default/lib/libexec:/run/current-system/sw/lib/libexec
WAYLAND_DISPLAY=wayland-0
PWD=/home/sei40kr
NIX_PATH=nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos:nixos-config=/etc/nixos/configuration.nix:/nix/var/nix/profiles/per-user/root/channels
HOME=/home/sei40kr
LC_CTYPE=ja_JP.UTF-8
JOURNAL_STREAM=8:1665465
XDG_SESSION_TYPE=wayland
GTK_PATH=/home/sei40kr/.nix-profile/lib/gtk-2.0:/home/sei40kr/.nix-profile/lib/gtk-3.0:/etc/profiles/per-user/sei40kr/lib/gtk-2.0:/etc/profiles/per-user/sei40kr/lib/gtk-3.0:/nix/var/nix/profiles/default/lib/gtk-2.0:/nix/var/nix/profiles/default/lib/gtk-3.0:/run/current-system/sw/lib/gtk-2.0:/run/current-system/sw/lib/gtk-3.0
XDG_DATA_DIRS=/nix/store/q29xkbcc56wq1iqwp7qcwjmswywqiv8b-shared-mime-info-2.1/share:/nix/store/00sf5xsg3i6s8ggnflfzanbgv71hl70f-gnome-shell-40.1/share:/nix/store/97y7kws9r7frjk7y6z4ibvgbnzdqd350-gtk+3-3.24.27/share/gsettings-schemas/gtk+3-3.24.27:/nix/store/41aa9xzsbjfgfk710sn5fg4xgzq6j6pc-gsettings-desktop-schemas-40.0/share/gsettings-schemas/gsettings-desktop-schemas-40.0:/nix/store/g33rhgs45y8fdx6cy6ki0qpmvdvpwi9d-gnome-keyring-40.0/share/gsettings-schemas/gnome-keyring-40.0:/nix/store/qpdw6wvg6nyffaxs44yp94l0iq3jg3f3-gcr-3.40.0/share/gsettings-schemas/gcr-3.40.0:/nix/store/liabvc30rxzs4fkk6wv5cglq50wxvjmm-mutter-40.1/share/gsettings-schemas/mutter-40.1:/nix/store/d0lavdiy4jhqi9ih735xh29smjlbjydl-evolution-data-server-3.40.2/share/gsettings-schemas/evolution-data-server-3.40.2:/nix/store/bqcd5zrbbqr1xm62f4qczgwiqshcya1c-gtk4-4.2.1/share/gsettings-schemas/gtk4-4.2.1:/nix/store/nmwscxrmyxsx9mh4jmbqmkpx69pwxcxj-gdm-40.0/share/gsettings-schemas/gdm-40.0:/nix/store/q78bhi0r2f2srrbhpq875zg9gfn8xc6f-gnome-clocks-40.0/share/gsettings-schemas/gnome-clocks-40.0:/nix/store/3yq2ivk5jyx4n31r89s4drkanzviimsm-ibus-1.5.24/share/gsettings-schemas/ibus-1.5.24:/nix/store/590q9zn94bgh0b7wr9a10w3hyzxmh4ab-gnome-settings-daemon-40.0.1/share/gsettings-schemas/gnome-settings-daemon-40.0.1:/nix/store/i047bkk6rrr80968h8znc8r6g1qqx1c8-libgweather-40.0/share/gsettings-schemas/libgweather-40.0:/nix/store/w72h31n364ckzfaa9xn0fcrq06vy16wc-libnma-1.8.30/share/gsettings-schemas/libnma-1.8.30:/nix/store/00sf5xsg3i6s8ggnflfzanbgv71hl70f-gnome-shell-40.1/share/gsettings-schemas/gnome-shell-40.1:/nix/store/97y7kws9r7frjk7y6z4ibvgbnzdqd350-gtk+3-3.24.27/share:/nix/store/97za2120lnkx0n5vdpk22ij54y8mm22h-cups-2.3.3/share:/nix/store/qpdw6wvg6nyffaxs44yp94l0iq3jg3f3-gcr-3.40.0/share:/nix/store/bqcd5zrbbqr1xm62f4qczgwiqshcya1c-gtk4-4.2.1/share:/nix/store/fnc10mb9f58fipvlm3i69l3f8c2rmw5z-adwaita-icon-theme-40.1.1/share:/nix/store/n0snbh1y2g99grfpi7wf4r9lp6wacp62-hicolor-icon-theme-0.17/share:/nix/store/0bnqia6yjdypx66pc4phj0bkwd4l6q9c-gnome-bluetooth-3.34.5/share:/nix/store/q78bhi0r2f2srrbhpq875zg9gfn8xc6f-gnome-clocks-40.0/share:/nix/store/3yq2ivk5jyx4n31r89s4drkanzviimsm-ibus-1.5.24/share:/nix/store/94l6fyhfav1lss5b9s1f2ss7hkhq7pjn-desktops/share:/home/sei40kr/.nix-profile/share:/etc/profiles/per-user/sei40kr/share:/nix/var/nix/profiles/default/share:/run/current-system/sw/share:/nix/store/17s1r7x7w93lsvhjkwa3d5sfr545fznd-gnome-session-40.1.1/share:/nix/store/41aa9xzsbjfgfk710sn5fg4xgzq6j6pc-gsettings-desktop-schemas-40.0/share/gsettings-schemas/gsettings-desktop-schemas-40.0:/nix/store/97y7kws9r7frjk7y6z4ibvgbnzdqd350-gtk+3-3.24.27/share/gsettings-schemas/gtk+3-3.24.27:/nix/store/590q9zn94bgh0b7wr9a10w3hyzxmh4ab-gnome-settings-daemon-40.0.1/share/gsettings-schemas/gnome-settings-daemon-40.0.1:/nix/store/17s1r7x7w93lsvhjkwa3d5sfr545fznd-gnome-session-40.1.1/share/gsettings-schemas/gnome-session-40.1.1:/nix/store/00sf5xsg3i6s8ggnflfzanbgv71hl70f-gnome-shell-40.1/share:/nix/store/17s1r7x7w93lsvhjkwa3d5sfr545fznd-gnome-session-40.1.1/share:/nix/store/41aa9xzsbjfgfk710sn5fg4xgzq6j6pc-gsettings-desktop-schemas-40.0/share/gsettings-schemas/gsettings-desktop-schemas-40.0:/nix/store/97y7kws9r7frjk7y6z4ibvgbnzdqd350-gtk+3-3.24.27/share/gsettings-schemas/gtk+3-3.24.27:/nix/store/590q9zn94bgh0b7wr9a10w3hyzxmh4ab-gnome-settings-daemon-40.0.1/share/gsettings-schemas/gnome-settings-daemon-40.0.1:/nix/store/17s1r7x7w93lsvhjkwa3d5sfr545fznd-gnome-session-40.1.1/share/gsettings-schemas/gnome-session-40.1.1:/nix/store/00sf5xsg3i6s8ggnflfzanbgv71hl70f-gnome-shell-40.1/share
GI_TYPELIB_PATH=/nix/store/97y7kws9r7frjk7y6z4ibvgbnzdqd350-gtk+3-3.24.27/lib/girepository-1.0:/nix/store/wx2kqalxzqhax1xfhjnnyr3fvd8m1ibm-librsvg-2.50.6/lib/girepository-1.0:/nix/store/41aa9xzsbjfgfk710sn5fg4xgzq6j6pc-gsettings-desktop-schemas-40.0/lib/girepository-1.0:/nix/store/gx2gg5xq45fzyykyxbv8xyv2ci3zpn49-atk-2.36.0/lib/girepository-1.0:/nix/store/ahk5g9mdra2vqb3nk8v39pijsb4hjycs-gdk-pixbuf-2.42.6/lib/girepository-1.0:/nix/store/azw8ixqngmrlaln4309fsw8ps0ckmz37-harfbuzz-2.8.1/lib/girepository-1.0:/nix/store/rywq4m4ad9gslv4r41mzyply6yal234m-pango-1.48.4/lib/girepository-1.0:/nix/store/qpdw6wvg6nyffaxs44yp94l0iq3jg3f3-gcr-3.40.0/lib/girepository-1.0:/nix/store/6qfs4c9l69kkmpkslz4yw0l0p7hmzar1-accountsservice-0.6.55/lib/girepository-1.0:/nix/store/gkq59s1z3f2ls09cwwzhv53cqnmq0416-libsecret-0.20.4/lib/girepository-1.0:/nix/store/7w7s5n0hxfjr6lxxpr6lzprhfzm9i8hl-polkit-0.118/lib/girepository-1.0:/nix/store/5vpskk2jxy3iicy30w5qy7a5m11fgxcg-networkmanager-1.30.4/lib/girepository-1.0:/nix/store/g2dksmm93fd0yazb60wam4fx7qiy0am0-json-glib-1.6.2/lib/girepository-1.0:/nix/store/z6gxbwzya2k24fs7qhrbc5ffjd82vw1x-graphene-1.10.6/lib/girepository-1.0:/nix/store/n9vw2c3i62rwpwa5p97an116nn66j4wz-libical-3.0.10/lib/girepository-1.0:/nix/store/7rfv5b6626vyy671i44r710nysammfqh-libsoup-2.72.0/lib/girepository-1.0:/nix/store/d0lavdiy4jhqi9ih735xh29smjlbjydl-evolution-data-server-3.40.2/lib/girepository-1.0:/nix/store/bqcd5zrbbqr1xm62f4qczgwiqshcya1c-gtk4-4.2.1/lib/girepository-1.0:/nix/store/nmwscxrmyxsx9mh4jmbqmkpx69pwxcxj-gdm-40.0/lib/girepository-1.0:/nix/store/hfv5r4dz09d3y3znw88m3a04pr80iwjp-geoclue-2.5.7/lib/girepository-1.0:/nix/store/0bnqia6yjdypx66pc4phj0bkwd4l6q9c-gnome-bluetooth-3.34.5/lib/girepository-1.0:/nix/store/ac099cg1jgpb5fyp1i1yy6c4jzx0c8kf-at-spi2-core-2.40.1/lib/girepository-1.0:/nix/store/4k5f3m5zjnivhnwrdmig8b8ggklp1270-upower-0.99.11/lib/girepository-1.0:/nix/store/3yq2ivk5jyx4n31r89s4drkanzviimsm-ibus-1.5.24/lib/girepository-1.0:/nix/store/9j2qngbhlsqr9in504snbcwxkbry1iiw-gnome-desktop-40.1/lib/girepository-1.0:/nix/store/xkz01kz5namrc5q617ard9vz7pxk6h15-gobject-introspection-1.68.0/lib/girepository-1.0:/nix/store/7ywx7iqg40m0qcm2a4jqbqr71hx6kq71-gstreamer-1.18.4/lib/girepository-1.0:/nix/store/gnkyaz0bp4ppgfh69gp9acnz6dxinpba-gst-plugins-base-1.18.4/lib/girepository-1.0:/nix/store/i047bkk6rrr80968h8znc8r6g1qqx1c8-libgweather-40.0/lib/girepository-1.0:/nix/store/w72h31n364ckzfaa9xn0fcrq06vy16wc-libnma-1.8.30/lib/girepository-1.0:/nix/store/24bd8pw7ajbbag6f719v0q3z053034xv-gnome-autoar-0.3.2/lib/girepository-1.0
XDG_SESSION_DESKTOP=gnome
GIO_EXTRA_MODULES=/nix/store/w5jii7aib2ysjcbaj7fcc1g6h38sq5xz-glib-networking-2.68.1/lib/gio/modules:/nix/store/wpvj3wjm1kzxygnclsybdx5c44wld8h5-dconf-0.40.0-lib/lib/gio/modules:/nix/store/w5jii7aib2ysjcbaj7fcc1g6h38sq5xz-glib-networking-2.68.1/lib/gio/modules:/nix/store/wpvj3wjm1kzxygnclsybdx5c44wld8h5-dconf-0.40.0-lib/lib/gio/modules
GJS_DEBUG_OUTPUT=stderr
TERMINFO_DIRS=/home/sei40kr/.nix-profile/share/terminfo:/etc/profiles/per-user/sei40kr/share/terminfo:/nix/var/nix/profiles/default/share/terminfo:/run/current-system/sw/share/terminfo
GDK_PIXBUF_MODULE_FILE=/nix/store/wx2kqalxzqhax1xfhjnnyr3fvd8m1ibm-librsvg-2.50.6/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
QT_PLUGIN_PATH=/home/sei40kr/.nix-profile/lib/qt4/plugins:/home/sei40kr/.nix-profile/lib/kde4/plugins:/etc/profiles/per-user/sei40kr/lib/qt4/plugins:/etc/profiles/per-user/sei40kr/lib/kde4/plugins:/nix/var/nix/profiles/default/lib/qt4/plugins:/nix/var/nix/profiles/default/lib/kde4/plugins:/run/current-system/sw/lib/qt4/plugins:/run/current-system/sw/lib/kde4/plugins
TERM=alacritty
SHELL=/run/current-system/sw/bin/zsh
XDG_SESSION_CLASS=user
GNOME_SETUP_DISPLAY=:1
XDG_CURRENT_DESKTOP=GNOME
SHLVL=1
LOCALE_ARCHIVE=/run/current-system/sw/lib/locale/locale-archive
XDG_DESKTOP_PORTAL_DIR=/nix/store/6pa52gc42pkdmqyh9a8i2ihgg6f5g0l6-xdg-portals/share/xdg-desktop-portal/portals
GDMSESSION=gnome
LOGNAME=sei40kr
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
XDG_RUNTIME_DIR=/run/user/1000
XAUTHORITY=/run/user/1000/.mutter-Xwaylandauth.0K2G60
XDG_CONFIG_DIRS=/home/sei40kr/.nix-profile/etc/xdg:/etc/profiles/per-user/sei40kr/etc/xdg:/nix/var/nix/profiles/default/etc/xdg:/run/current-system/sw/etc/xdg:/nix/store/590q9zn94bgh0b7wr9a10w3hyzxmh4ab-gnome-settings-daemon-40.0.1/etc/xdg:/nix/store/590q9zn94bgh0b7wr9a10w3hyzxmh4ab-gnome-settings-daemon-40.0.1/etc/xdg
PATH=/run/wrappers/bin:/home/sei40kr/.nix-profile/bin:/etc/profiles/per-user/sei40kr/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
INFOPATH=/home/sei40kr/.nix-profile/info:/home/sei40kr/.nix-profile/share/info:/etc/profiles/per-user/sei40kr/info:/etc/profiles/per-user/sei40kr/share/info:/nix/var/nix/profiles/default/info:/nix/var/nix/profiles/default/share/info:/run/current-system/sw/info:/run/current-system/sw/share/info
GDM_X_SERVER_EXTRA_ARGS=-config /nix/store/vwn673c2cir2hqrgkcda5kg9pll5iy1q-xserver.conf -xkbdir /nix/store/lnr53k4fkc2kvilwj428r3z7n3a1w3h4-xkeyboard-config-2.33/etc/X11/xkb -logfile /dev/null -nolisten tcp
GJS_DEBUG_TOPICS=JS ERROR;JS LOG
GST_PLUGIN_SYSTEM_PATH_1_0=/nix/store/i9q8bj3ghrjnqmhignd6srxqx6m2vrhf-pipewire-0.3.30-lib/lib/gstreamer-1.0:/nix/store/7ywx7iqg40m0qcm2a4jqbqr71hx6kq71-gstreamer-1.18.4/lib/gstreamer-1.0:/nix/store/gnkyaz0bp4ppgfh69gp9acnz6dxinpba-gst-plugins-base-1.18.4/lib/gstreamer-1.0:/nix/store/lf393q0rjkd8i0ci9nzc6yrk8jghc6f3-gst-plugins-good-1.18.4/lib/gstreamer-1.0:/nix/store/i9q8bj3ghrjnqmhignd6srxqx6m2vrhf-pipewire-0.3.30-lib/lib/gstreamer-1.0:/nix/store/7ywx7iqg40m0qcm2a4jqbqr71hx6kq71-gstreamer-1.18.4/lib/gstreamer-1.0:/nix/store/gnkyaz0bp4ppgfh69gp9acnz6dxinpba-gst-plugins-base-1.18.4/lib/gstreamer-1.0:/nix/store/lf393q0rjkd8i0ci9nzc6yrk8jghc6f3-gst-plugins-good-1.18.4/lib/gstreamer-1.0
SESSION_MANAGER=local/nixos:@/tmp/.ICE-unix/315074,unix/nixos:/tmp/.ICE-unix/315074
XCURSOR_PATH=/home/sei40kr/.icons:/home/sei40kr/.local/share/icons:/home/sei40kr/.nix-profile/share/icons:/home/sei40kr/.nix-profile/share/pixmaps:/etc/profiles/per-user/sei40kr/share/icons:/etc/profiles/per-user/sei40kr/share/pixmaps:/nix/var/nix/profiles/default/share/icons:/nix/var/nix/profiles/default/share/pixmaps:/run/current-system/sw/share/icons:/run/current-system/sw/share/pixmaps
_=/run/current-system/sw/bin/env

And then I noticed none of GTK_IM_MODULE, QT_IM_MODULE was set.
Wayland session doesn't read /etc/profile, so we may need those environment variables in /etc/environment.
Thank you for your help.

I put this in my configuration:
{
    environment.sessionVariables = {
      GTK_IM_MODULE = "fcitx";
      QT_IM_MODULE = "fcitx";
      XMODIFIERS = "@im=fcitx";
    };
}

and confirmed those variables in set in GNOME session. (same way as above)
But it's still not working. There probably exists another cause.

It's also not working for me in Plasma5 / Wayland, even if I set those env variables.

Exporting NIX_USER_PROFILE_DIR and NIX_PROFILES worked for me:
sh-4.4$ export NIX_USER_PROFILE_DIR="/nix/var/nix/profiles/per-user/$USER"
sh-4.4$ export NIX_PROFILES="/run/current-system/sw /nix/var/nix/profiles/default /etc/profiles/per-user/$USER $HOME/.nix-profile"
sh-4.4$ google-chrome-stable

I think those variables have nothing to do with Fcitx 5, but somehow, it works :thinking:

https://github.com/NixOS/nixpkgs/blob/42cadf5a768c81d912adba037d30c39d09f326c7/pkgs/development/libraries/gtk/patches/3.0-immodules.cache.patch#L8
May be this is related.

    environment.sessionVariables = {
      NIX_PROFILES =
        "${concatStringsSep " " (reverseList config.environment.profiles)}";
      GTK_IM_MODULE = "fcitx";
      QT_IM_MODULE = "fcitx";
      XMODIFIERS = "@im=fcitx";
    };

This works great for me!

Use environment.sessionVariables to export GTK_IM_MODULE, QT_IM_MODULE, XMODIFIERS because Wayland session doesn't read set-environment
Export NIX_PROFILES, NIX_USER_PROFILE_DIRS, and other Nix basic variables in pam-environment

Please check https://github.com/fcitx/fcitx5/issues/292#issuecomment-864396920.
Wayland support is in progress and you may need xwayland now.
On 2021年7月8日 at 06:19:20, Seong Yong-ju @.@.)) wrote:

environment.sessionVariables = { NIX_PROFILES = "${concatStringsSep " " (reverseList config.environment.profiles)}"; GTK_IM_MODULE = "fcitx"; QT_IM_MODULE = "fcitx"; XMODIFIERS = @.***=fcitx"; };
This works great for me!
Use environment.sessionVariables to export GTK_IM_MODULE, QT_IM_MODULE, XMODIFIERS because Wayland session doesn't read set-environment
Export NIX_PROFILES, NIX_USER_PROFILE_DIRS, and other Nix basic variables in pam-environment
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub(https://github.com/NixOS/nixpkgs/issues/129442#issuecomment-875972207), or unsubscribe(https://github.com/notifications/unsubscribe-auth/AA2BI77XIZ7YSRKDHP5LNDLTWTHGRANCNFSM474XCIKA).

Setting those variables to environment.sessionVariables does indeed seem to work. The Fcitx5 module in NixOS already sets the *_IM_MODULE variables... but it does that using environment.variables, which should probably be changed. Does environment.sessionVariables work in X11 environments? Not sure what the right place for setting NIX_PROFILES would be but I'm pretty sure it's not in the Fcitx5 module.

The default environment.variables is set to sessionVariables.
And environment.variables is run in /etc/profile with set-environment script
defined by  <nixpkgs/nixos/modules/config/shells-environment.nix>.
This file is run in xsession-wrapper defined by
<nixpkgs/nixos/modules/services/x11/display-managers/default.nix>,
before desktop-manager is started.
According to documents, environment.sessionVariables is set by PAM
implemented in <nixpkgs/nixos/modules/config/system-environment>, and
it is merged into environment.variables. Only a little variables is set
in it.
NIX_PROFILES is set in environment.extraInit
(<nixpkgs/nixos/modules/programs/environment.nix) now.
I don't think it is a good idea to move these into sessionVariables.
jansol @.***> writes:

Setting those variables to environment.sessionVariables does indeed seem to work. The Fcitx5 module in NixOS already sets the *_IM_MODULE variables... but it does that using environment.variables, which should probably be changed. Does environment.sessionVariables work in X11 environments? Not sure what the right place for setting NIX_PROFILES would be but I'm pretty sure it's not in the Fcitx5 module.

I'd like to let wayland use set-environment rather than move them into sessionvariables. Is there a startup script for wayland  like xprofile?

@Vonfry

Is there a startup script for wayland like xprofile?

Basically, no.
We have 2 ways to set environment for Wayland session:

Write in pam-environment & let display manager read it
Make a wrapper for desktop manager (just like extraSessionCommands in Sway)

I perfer to make a wrapper.
Currently, x11 is also worked with a wrapper. You can check
xsessionWrapper in <nixpkgs/nixos/modules/services/x11/display-manager/default.nix>.
Then desktop-manager is also worked with a wrapper in
services.xserver.desktopManager.session.start
(<nixpkgs/nixos/modules/services/x11/desktop-managers/default.nix>).
Seong Yong-ju @.***> writes:

@Vonfry

Is there a startup script for wayland like xprofile?

Basically, no.
We have 2 ways to set environment for Wayland session:

Write in pam-environment & let display manager read it
Make a wrapper for desktop manager (just like extraSessionCommands in Sway)

Sorry if this is entirely a different issue (it probably is), but I can't even figure out how to get fcitx5 to recognize fcitx5-mozc.
Using fcitx5-configtools doesn't even show mozc for me:

I do have the following, so I don't quite get how this could happen.
{
  i18n.inputMethod = {
    enabled = "fcitx5";
    fcitx5.addons = with pkgs; [ fcitx5-mozc ];
  };
}

I'm on:
$ nixos-version
22.05pre352272.c5051e2b5fe (Quokka)

You should search mozc instead of japanese, because the name of mozc
doesn't involve japanese. By the way, you may also uncheck "Only Show
Current Language".
Sage Hane @.***> writes:

Sorry if this is entirely a different issue (it probably is), but I can't even figure out how to get fcitx5 to recognize fcitx5-mozc.
Using fcitx5-configtools doesn't even show mozc for me:
image
I do have the following, so I don't quite get how this could happen.
{
i18n.inputMethod = {
enabled = "fcitx5";
fcitx5.addons = with pkgs; [ fcitx5-mozc ];
};
}
I'm on:
$ nixos-version
22.05pre352272.c5051e2b5fe (Quokka)
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.

I've tried that too. The issue seems to be that the wrong Addon Config Dir was being read, so I opened an issue at https://github.com/NixOS/nixpkgs/issues/160437.
I do find it weird that nobody else other than me seems to have encountered it. I wonder if I'm doing something wrong.

@sagehane
If you can't find Mozc by searching mozc, then restart Fcitx5 daemon by running
fcitx5 -r

and try again.
If this won't fix your issue, it's entirely a different issue.

Yep, I've tried plenty of stuff, including manually doing fcitx-remote -s, it's a different issue from my understanding. Sorry to bother.

    environment.sessionVariables = {
      NIX_PROFILES =
        "${concatStringsSep " " (reverseList config.environment.profiles)}";
      GTK_IM_MODULE = "fcitx";
      QT_IM_MODULE = "fcitx";
      XMODIFIERS = "@im=fcitx";
    };

This works great for me!
1. Use `environment.sessionVariables` to export `GTK_IM_MODULE`, `QT_IM_MODULE`, `XMODIFIERS` because Wayland session doesn't read `set-environment`

2. Export `NIX_PROFILES`, `NIX_USER_PROFILE_DIRS`, and other Nix basic variables in `pam-environment`

I just hit this problem myself after switching to Wayland on KDE Plasma. Thanks for this snippet! Until a proper solution for Wayland is developed, I will be using this configuration workaround.

    environment.sessionVariables = {
      NIX_PROFILES =
        "${concatStringsSep " " (reverseList config.environment.profiles)}";
      GTK_IM_MODULE = "fcitx";
      QT_IM_MODULE = "fcitx";
      XMODIFIERS = "@im=fcitx";
    };

This works great for me!

Use environment.sessionVariables to export GTK_IM_MODULE, QT_IM_MODULE, XMODIFIERS because Wayland session doesn't read set-environment
Export NIX_PROFILES, NIX_USER_PROFILE_DIRS, and other Nix basic variables in pam-environment

Saved my day! Thanks!

FWIW with the plasma6 module you don't need to set any environment variables for Fcitx5 to work. In fact it will yell at you if you do.
HOWEVER if you are switching over from an older version or from X11 you should make sure to delete all KDE and Fcitx-related files and subdirectories from your ~/.config as those probably still try to launch the X11 version of Fcitx5

And you'll also need to enable the Fcitx 5 virtual keyboard in Plasma settings under Keyboard > Virtual Keyboard (technically you should enable Fcitx 5 Wayland Launcher but it seems to revert to Fcitx 5 afterwards and still work fine:


save trajectory

Hello,
Is it possible to save trajectory in some convenient way? Or should I just write a node to save the topic in a txt file?
Regards

Though we are a little late to respond, we encourage such user-specific issues to be raised on the Knowledge hub, so that our experienced mentors can chime in and new students can also learn from the discussion thread.
Thank you


do not pull image for 10m after error 421

What will be the behavior in such a case? When will Kubevuln resume pulling the images?

Next time it is called by operator, but only if the timeout expired


Any suggestions for the HPC requirement

Really appreciate the delicate project and codes. I am running the code and find it stuck in the preprocessing phase due to the lack of RAM. My workstation is 128G RAM with 4 RTX 3090. Could you help suggest the minimum requirement for processing the data, e.g., norman dataset?
Thanks!

@yhr91 The program is killed in "prepare_split()" with "simulation" mode. Thanks.

Hi, thanks for your question! We mainly used Nvidia 2080 Ti RTX 11GB GPUs to train GEARS. The node we used had 768GB of RAM.
Having said that, I think the RAM you have should be sufficient for training the model. I believe GEARS uses under 15GB of RAM during training on Norman. Are you sure you don't have other processes running in the background?

Hi, thanks for your question! We mainly used Nvidia 2080 Ti RTX 11GB GPUs to train GEARS. The node we used had 768GB of RAM.
Having said that, I think the RAM you have should be sufficient for training the model. I believe GEARS uses under 15GB of RAM during training on Norman. Are you sure you don't have other processes running in the background?

Thanks for your prompt response. Yes I have checked the RAM usage of this one process, and it really took over 128G after a while. It stopped at the "prepare_split()" with the "simulation" mode. May I have the versions of the libraries in the requirements? This issue may arise from the library versions...

Hi, thanks for your question! We mainly used Nvidia 2080 Ti RTX 11GB GPUs to train GEARS. The node we used had 768GB of RAM.
Having said that, I think the RAM you have should be sufficient for training the model. I believe GEARS uses under 15GB of RAM during training on Norman. Are you sure you don't have other processes running in the background?

The memory usage of more than 128G happens in the preprocessing period... Could you please help check if this also happens to your node? Many thanks!

I have resolved this issue by upgrading the version of scanpy and pandas... Thanks for the help.


Fyle import attributes

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (master@b8487cc). Click here to learn what that means.

Additional details and impacted files
@@            Coverage Diff            @@
##             master      #32   +/-   ##
=========================================
  Coverage          ?   87.65%
=========================================
  Files             ?       30
  Lines             ?     1207
  Branches          ?        0
=========================================
  Hits              ?     1058
  Misses            ?      149
  Partials          ?        0

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Support whitespace after right square bracket in structured lg

fix: #2836

Pull Request Test Coverage Report for Build 85424

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 77.607%

Totals

Change from base Build 85358:
0.0%

Covered Lines:
5992

Relevant Lines:
7721

💛  - Coveralls

:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Builder.dll compared against version 4.5.3
:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Builder.AI.Luis.dll compared against version 4.5.3
:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Builder.AI.QnA.dll compared against version 4.5.3
:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Builder.ApplicationInsights.dll compared against version 4.5.3
:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Builder.Azure.dll compared against version 4.5.3
:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Builder.Dialogs.dll compared against version 4.5.3
:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Builder.TemplateManager.dll compared against version 4.5.3
:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Configuration.dll compared against version 4.5.3
:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Connector.dll compared against version 4.5.3
:heavy_check_mark: No Binary Compatibility issues for Microsoft.Bot.Schema.dll compared against version 4.5.3