[Errno 17] File exists when extracting files to egg cache (or creating egg cache directory)

We are experiencing intermittent failures when running tests for one of our Python applications while installing dependencies during test execution in our CI system. The tests are being run with pytest, and we are using the pytest-xdist plugin to run tests in multiple processes in parallel to reduce the execution time.
Here is an example stack trace:
.eggs/py-1.5.2-py2.7.egg/py/_path/common.py:377: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
.eggs/py-1.5.2-py2.7.egg/py/_path/common.py:429: in gen
    for p in self.gen(subdir):
.eggs/py-1.5.2-py2.7.egg/py/_path/common.py:419: in gen
    if p.check(dir=1) and (rec is None or rec(p))])
.eggs/pytest-3.2.5-py2.7.egg/_pytest/main.py:737: in _recurse
    ihook = self.gethookproxy(path)
.eggs/pytest-3.2.5-py2.7.egg/_pytest/main.py:641: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
.eggs/pytest-3.2.5-py2.7.egg/_pytest/config.py:356: in _getconftestmodules
    mod = self._importconftest(conftestpath)
.eggs/pytest-3.2.5-py2.7.egg/_pytest/config.py:381: in _importconftest
    raise ConftestImportFailure(conftestpath, sys.exc_info())
E   ConftestImportFailure: ExtractionError("Can't extract file(s) to egg cache\n\nThe following error occurred while trying to extract file(s)\nto the Python egg cache:\n\n  [Errno 17] File exists: '/home/travis/.cache/Python-Eggs'\n\nThe Python egg cache directory is currently set to:\n\n  /home/travis/.cache/Python-Eggs\n\nPerhaps your account does not have write access to this directory?\nYou can change the cache directory by setting the PYTHON_EGG_CACHE\nenvironment variable to point to an accessible directory.\n",)
E     File "/home/travis/build/finn/fiaas-deploy-daemon/.eggs/pytest-3.2.5-py2.7.egg/_pytest/assertion/rewrite.py", line 212, in load_module
E       py.builtin.exec_(co, mod.__dict__)
E     File "/home/travis/build/finn/fiaas-deploy-daemon/.eggs/py-1.5.2-py2.7.egg/py/_builtin.py", line 221, in exec_
E       exec2(obj, globals, locals)
E     File "<string>", line 7, in exec2
E     File "/home/travis/build/finn/fiaas-deploy-daemon/tests/fiaas_deploy_daemon/conftest.py", line 9, in <module>
E       from fiaas_deploy_daemon.specs.models import AppSpec, ResourceRequirementSpec, ResourcesSpec, PrometheusSpec, \
E     File "/home/travis/build/finn/fiaas-deploy-daemon/fiaas_deploy_daemon/__init__.py", line 13, in <module>
E       from .crd import CustomResourceDefinitionBindings, DisabledCustomResourceDefinitionBindings
E     File "/home/travis/build/finn/fiaas-deploy-daemon/fiaas_deploy_daemon/crd/__init__.py", line 7, in <module>
E       from .watcher import CrdWatcher
E     File "/home/travis/build/finn/fiaas-deploy-daemon/fiaas_deploy_daemon/crd/watcher.py", line 12, in <module>
E       from ..deployer import DeployerEvent
E     File "/home/travis/build/finn/fiaas-deploy-daemon/fiaas_deploy_daemon/deployer/__init__.py", line 8, in <module>
E       from .deploy import Deployer
E     File "/home/travis/build/finn/fiaas-deploy-daemon/fiaas_deploy_daemon/deployer/deploy.py", line 7, in <module>
E       from .kubernetes.ready_check import ReadyCheck
E     File "/home/travis/build/finn/fiaas-deploy-daemon/fiaas_deploy_daemon/deployer/kubernetes/__init__.py", line 7, in <module>
E       from .adapter import K8s
E     File "/home/travis/build/finn/fiaas-deploy-daemon/fiaas_deploy_daemon/deployer/kubernetes/adapter.py", line 9, in <module>
E       from ...specs.models import ResourcesSpec, ResourceRequirementSpec
E     File "/home/travis/build/finn/fiaas-deploy-daemon/fiaas_deploy_daemon/specs/__init__.py", line 10, in <module>
E       from .app_config_downloader import AppConfigDownloader
E     File "/home/travis/build/finn/fiaas-deploy-daemon/fiaas_deploy_daemon/specs/app_config_downloader.py", line 1, in <module>
E       import yaml
E     File "build/bdist.linux-x86_64/egg/yaml/__init__.py", line 14, in <module>
E     File "build/bdist.linux-x86_64/egg/yaml/cyaml.py", line 5, in <module>
E     File "build/bdist.linux-x86_64/egg/_yaml.py", line 7, in <module>
E     File "build/bdist.linux-x86_64/egg/_yaml.py", line 4, in __bootstrap__
E     File "/home/travis/.local/share/virtualenvs/fiaas-deploy-daemon/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1226, in resource_filename
E       self, resource_name
E     File "/home/travis/.local/share/virtualenvs/fiaas-deploy-daemon/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1726, in get_resource_filename
E       self._extract_resource(manager, self._eager_to_zip(name))
E     File "/home/travis/.local/share/virtualenvs/fiaas-deploy-daemon/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1756, in _extract_resource
E       self.egg_name, self._parts(zip_path)
E     File "/home/travis/.local/share/virtualenvs/fiaas-deploy-daemon/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1293, in get_cache_path
E       self.extraction_error()
E     File "/home/travis/.local/share/virtualenvs/fiaas-deploy-daemon/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1273, in extraction_error
E       raise err

While I'm not that familiar with the inner workings of pkg_resources, I took a look at the code referenced at the end of the stack trace, and my theory is that since the tests run in parallel, multiple threads may end up calling _bypass_ensure_directory at the same time, and that the error stems from a file system race condition between isdir and mkdir when creating the directory, similar to in https://github.com/pypa/setuptools/pull/1083. Perhaps using the same solution could work here?

Thanks for the workaround, I've applied it to the application's build system and it looks good this far at least.

I feel like changing the default here only pushes the can down the road. I'm fine with making --always-unzip the default, but I don't think that properly addresses the two issues identified in this ticket.:

the race condition of isdir/mkdir for zip-safe packages.
improper detection of pyyaml as zip safe when it is not (detection of _yaml.py as a stub).

Isn't ignoring _yaml.py as a stub the correct (explicit) behavior?

I think the normal behavior of the stubs is meant for extension modules, when mod.pyo is the module name, then setuptools creates a mod.py as a stub. But in the case of _yaml.py, it's not a setuptools-created stub but instead a proper Python module supplied by the project, and it's only a coincidence that it happens to look like a setuptools-created stub. In any case, _yaml isn't an extension module but it's being detected as one, and that's the failure. And the reason it doesn't happen on Python 3 is because the .pyc is created with a different name than the module (including the .cpython-36 before the extension).

So there are multiple issues at play then. With the following sample project:

setup.py:

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("helloworld.pyx")
)

hellloworld.pyx:

print "Hello World"

And building an egg with python setup.py bdist_egg:

Python 2.7 marks the egg as zip-safe (build/bdist.linux-x86_64/egg/helloworld.pyc is correctly detected as a stub): correct behavior
but Python 3.6 marks the egg as not-zip-safe because build/bdist.linux-x86_64/egg/__pycache__/helloworld.cpython-36.pyc is not detected as a stub: bug, no?

I do believe that the introduction of __pycache__/{module}.{pyver}.pyc (PEP 3147) has violated these expectations. So yes, that does sound like the bug.

I'm about to submit a PR for this.
For google-ability and thread sanity, here's the error that I get for this from our CI system.
Funny enough, we've actually had this fixed for a year, but we never upstreamed the patch (sorry!) - We're making that wrong right today.
Sure, the zip safe thing is a good catch, but the real catch is that we've got a very edgecase where high volume CICD systems will hit this, and developers wont, which is why the community in general is not out with pitchforks. Since our codebase for the particular product we're hitting this on is so large, we have to use massive build machines, so we hit this race condition like clockwork.
07:08:44 Traceback (most recent call last):
07:08:44   File "bin/assembler.py", line 14, in <module>
07:08:44 Traceback (most recent call last):
07:08:44   File "bin/assembler.py", line 14, in <module>
07:08:44         from jinja2 import Template, FileSystemLoader, Environment
07:08:44 from jinja2 import Template, FileSystemLoader, Environment
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/__init__.py", line 33, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/__init__.py", line 33, in <module>
07:08:44   File "bin/assembler.py", line 14, in <module>
07:08:44     from jinja2 import Template, FileSystemLoader, Environment
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/__init__.py", line 33, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 13, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 13, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 13, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/nodes.py", line 19, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/nodes.py", line 19, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/nodes.py", line 19, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/utils.py", line 531, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/utils.py", line 531, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/utils.py", line 531, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg/markupsafe/__init__.py", line 292, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg/markupsafe/__init__.py", line 292, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg/markupsafe/__init__.py", line 292, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg/markupsafe/_speedups.py", line 7, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg/markupsafe/_speedups.py", line 7, in <module>
07:08:44   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg/markupsafe/_speedups.py", line 7, in <module>
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg/markupsafe/_speedups.py", line 4, in __bootstrap__
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg/markupsafe/_speedups.py", line 4, in __bootstrap__
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg/markupsafe/_speedups.py", line 4, in __bootstrap__
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1133, in resource_filename
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1133, in resource_filename
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1133, in resource_filename
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1639, in get_resource_filename
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1639, in get_resource_filename
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1639, in get_resource_filename
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1669, in _extract_resource
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1669, in _extract_resource
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1669, in _extract_resource
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1200, in get_cache_path
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1200, in get_cache_path
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1200, in get_cache_path
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1180, in extraction_error
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1180, in extraction_error
07:08:45   File "/home/jenkins/workspace/job-hidden/master/.build_python/lib/python2.7/site-packages/setuptools-39.2.0-py2.7.egg/pkg_resources/__init__.py", line 1180, in extraction_error
07:08:45 pkg_resources.ExtractionErrorpkg_resources.: ExtractionError: pkg_resources.ExtractionError: Can't extract file(s) to egg cache
07:08:45
07:08:45 The following error occurred while trying to extract file(s)
07:08:45 to the Python egg cache:
07:08:45
07:08:45   [Errno 17] File exists: '/home/jenkins/.cache/Python-Eggs'
07:08:45
07:08:45 The Python egg cache directory is currently set to:
07:08:45
07:08:45   /home/jenkins/.cache/Python-Eggs
07:08:45
07:08:45 Perhaps your account does not have write access to this directory?
07:08:45 You can change the cache directory by setting the PYTHON_EGG_CACHE
07:08:45 environment variable to point to an accessible directory.
07:08:45 Can't extract file(s) to egg cache


Stop setting PRODUCT_MODULE_NAME when not needed

Starlark memory reduction.

This breaks stuff in my testing. So not pushing it for the time, but I'll possibly get back to it later.


Fixing isochrone issues

Hey @ConnorStoneAstro,
There were some fairly significant bugs with both the creation of the isochrone grid and then selecting isochrones. Take a look at the changes I made in both get_isochrones.py and MIST_isochrones.py. From the testing I've done thus far it seems like I fixed these issues but since you wrote those parts definitely take a look and check if what I've done makes sens to you. In the meantime, I'm going to think about how to make the code more robust with either unit tests or better checks within the code itself.

I've created a new repo "light-house-analyses" where we can put jupyter notebooks. That way they wont interfere with the PRs here on the main repo. Could you move the notebooks over?

I've created a new repo "light-house-analyses" where we can put jupyter notebooks. That way they wont interfere with the PRs here on the main repo. Could you move the notebooks over?

That's a good idea! Yeah, let me work on moving everything over

Where is the "light-house-analyses" repo @ConnorStoneAstro ?

Where is the "light-house-analyses" repo @ConnorStoneAstro ?

I sent you an invitation to join the repo. Let me know if it doesn't show up in your inbox

Got it! Thanks!


Necessary Vault Policies

It would be nice to have documented the vault permissions needed for vaultingkube to work.
From my tests, it needs read to mounts and read and lists for the secrets, follows the rules:
{
  "path": {
      "sys/mounts": {
          "capabilities": [
            "read"
          ]
        }
    }
 }

{
  "path": {
      "${VK_VAULT_ROOT_MOUNT_PATH}/*": {
          "capabilities": [
            "read",
            "list"
          ]
        }
    }
}

OBS: Check the var!
Can't do the PR right now. At least there is some info here!

You're right, it should just be read and list.  I can eventually document this in the readme
https://github.com/sunshinekitty/vaultingkube/blob/master/comms/vault.go#L58
https://github.com/sunshinekitty/vaultingkube/blob/master/comms/vault.go#L95
https://github.com/sunshinekitty/vaultingkube/blob/master/comms/vault.go#L104
vaultingkube doesn't update/create anything on the Vault side, so nothing else here should be needed

Thanks!


TSM1: Panic on Grafana query

I am testing InfluxDB 0.9.5 Nightly with Sensu and Grafana, and have experienced a couple of InfluxDB crashes...
I put the first one down to user error, however have now had a repeat.
I am running: InfluxDB shell 0.9.5-nightly-66a31ee installed from nightly RPM on a CentOS 6.5 VM running in Parallels on OS X 10.11.
The log file is here: https://gist.github.com/fatmcgav/b997941c053a0f155160
Let me know if you need any further info.
Cheers
Gav

OK, so I just managed to reproduce the Panic by re-running the same query from Grafan...
So it's something about this query that is causing InfluxDB to panic:
SELECT mean(system) AS "system", mean(iowait) AS "iowait", mean("user") AS "user", mean(idle) AS "idle" FROM "graphite"."default".cpu WHERE host =~ /sensu*/ AND time > now() - 1d GROUP BY time(1m), host

OK, further testing suggest that it's the AND time > now() - 1d that's causing InfluxDB to panic...
If I go shorter, e.g.  - 1h, or - 15m then the query works.
Anything longer, and InfluxDB panic's...

[query] 2015/10/21 14:19:30 SELECT mean(system) AS "system", mean(iowait) AS "iowait", mean("user") AS "user", mean(idle) AS "idle" FROM "graphite"."default".cpu WHERE host =~ /sensu*/ AND time > now() - 1w GROUP BY time(5m), host
panic: decode of short block: got 1, exp 9

When you upgraded to the new nightly, did you delete your data and start with a clean DB?

@jwilder I haven't upgraded the nightly yet...

@fatmcgav I meant when you started running 66a31ee.  Were you running an 0.9.5 (or any) nightly before that?  If so, did you remove your data files before upgrading to it?

Ah, ok…
No, this was a fresh install with no previous data.

On 21 Oct 2015, at 17:28, Jason Wilder<EMAIL_ADDRESS>wrote:
@fatmcgav https://github.com/fatmcgav I meant when you started running 66a31ee. Were you running an 0.9.5 (or any) nightly before that? If so, did you remove your data files before upgrading to it?
—
Reply to this email directly or view it on GitHub https://github.com/influxdb/influxdb/issues/4521#issuecomment-149952402.

The panic looks like could be is an issue in the blocks that the TSM engine has written or the cursors.  A block to be decoded was too small so it panic'd.

This panic is caused by writing NaN float values.  #4587 should fix this.

Cheers for the quick fix. Will try the nightly tomorrow...
On 27 Oct 2015 19:22, "Jason Wilder"<EMAIL_ADDRESS>wrote:

This panic is caused by writing NaN float values. #4587
https://github.com/influxdb/influxdb/pull/4587 should fix this.
—
Reply to this email directly or view it on GitHub
https://github.com/influxdb/influxdb/issues/4521#issuecomment-151616375.

@fatmcgav Watch for #4587 to be merged.  It's not merged yet.

Ah, there's me trying to jump the gun... Will do, cheers...
On 27 Oct 2015 22:01, "Jason Wilder"<EMAIL_ADDRESS>wrote:

@fatmcgav https://github.com/fatmcgav Watch for #4587
https://github.com/influxdb/influxdb/pull/4587 to be merged. It's not
merged yet.
—
Reply to this email directly or view it on GitHub
https://github.com/influxdb/influxdb/issues/4521#issuecomment-151658355.


Implement the Mul trait

Implement the Mul trait to allow using the * operator between Mat, &Mat, or MatRef and a ComplexField type.

MulAssign in #49 is fine, though i'm not sure about Mul yet. i'd like to handle it on my own to make sure there aren't any conflicts with matrix multiplication. is it fine if i close this for now?

That is very reasonable.
Let me know if there are any "good first issue" level of difficulty things that you would like help with.


Rocket Chip Updates

This needs to be updated for the latest version of rocket-chip.  The TLBundle* classes no longer have error IO fields.
I applied a quick fix to that, but then I came across a Scala runtime error when trying to build the emulator.  The error is a java.lang.reflect.InvocationTargetException caused by:
java.lang.IllegalAccessError: tried to access field freechips.rocketchip.tile.RocketTileModuleImp.reflPoly$Cache4 from class freechips.rocketchip.tile.HasLazyRoCCModule$class
      at freechips.rocketchip.tile.HasLazyRoCCModule$class.reflMethod$Method4(LazyRoCC.scala:91)

The fix you mentions makes sense. I'm seeing the same error that you're seeing now.
That appears to be something actually wrong with Rocket as it's failing the RoccExampleConfig. I can dig into it, but once it starts to run into problems with Diplomacy (what this looks like) I get somewhat lost.

I went ahead and filed an issue on Rocket Chip here: https://github.com/freechipsproject/rocket-chip/issues/1515

As far as I can tell, this appears to be a Rocket issues. I'm playing around trying to see if I can figure out a fix there. Otherwise, you'll have to use a version of Rocket that is before this issue showed up.
The offending commit appears to be https://github.com/freechipsproject/rocket-chip/commit/c563c9ce83af70dfbd019aa1701d71b4ac60e700. However, it's not intuitive why this is breaking it.
I think that anything before that PR got merged should be fine.
I'm closing this, but feel free to re-open if you find that this is an issue with LeChiffre.

Thanks, moving to the commit before that got it to work.

This should be resolved with: https://github.com/freechipsproject/rocket-chip/commit/c07ae80590416043149e0e3ecb6f9217d074f8be


Show lengend and imprecision in map view

I guess this is a tricky one, but I don't want it to get lost: Many participants of our workshop mentioned that they would like to see the coordinate imprecision for individual records. Probably this could be done only for large zoom levels, and I'm not sure how this would look like.
Also, for many users it wasn't obvious that the color of the dots represents the number of occurrences in the cluster - so adding a small legend on what the colors mean would be nice.

+1 for this, thank you!!

I do not believe we will be able to show uncertainties and foot print WKTs in the occurrence search map. But we should and could show them for individual occurrences. At least that is the conclusion and implementation we have on GBIF.org

This is awesome Morten!! Thank you so much!!


Sending a transaction does not show in history

Shows in my balance on the Overview page, but not in the recent transactions or in the history.
So I did a rescan and the seeming missing transaction now shows up, but it has the wrong amount and the wrong hash.  Showed the correct working hash at the transaction sent dialog though.

This has been solved at #35


False positive when all capabilities dropped

Running kubeaudit caps returns a lot of ERRO[0003] Capability not dropped messages for the pods with the following effective settings:
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - all
      readOnlyRootFilesystem: true

From inside the container everything is clearly OK:
grep ^Cap /proc/1/status
CapInh:<PHONE_NUMBER>000000
CapPrm:<PHONE_NUMBER>000000
CapEff:<PHONE_NUMBER>000000
CapBnd:<PHONE_NUMBER>000000
CapAmb:<PHONE_NUMBER>000000

Thanks for reporting this 👍


fix(client): stop over-emitting spaces changes

Current dependencies on/for this PR:

main

PR #2841   👈

This comment was auto-generated by Graphite.


DefaultResourceReader bug

Fixing DefaultResourceReader bug when embedded resource has no extension.

This was solved in a different pull request. Thank you for taking the time to offer a solution!


rust: cargo: needs to handle warning in output (non-JSON)

Error screen

Steps to reproduce

modify ~/.cargo/config to switch registry
[source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index"
replace-with = 'ustc'
[source.ustc]
registry = "https://mirrors.ustc.edu.cn/crates.io-index/"

use 'cargo new' to create an empty binary project

add some dependencies in Cargo.toml

open main.rs in gVim and save it

Problem location

Line 43: because of cargo warnings, failed to decode json

I think warnings need to be filtered from the output, before parsing it as JSON.
This gets done with output from the actual job already, but this appears to happen during setup of the workspace.

Do the warnings and JSON come on different streams (e.g. stdout vs stderr ?). That would be a convenient way to filter.

@CameronNemo
It is just using "system()" there, so you do not get info about the stream.
Could use the same method as with https://github.com/neomake/neomake/blob/196158b2c6765dae69f382a1cbe1e94eac513400/autoload/neomake/makers/ft/rust.vim#L128-L130 probably (i.e. only handle output lines starting with "{").


Convert to CoreML (iOS)

Can anyone provide insights on the steps to convert the trained models to CoreML? Any help would be really appreciated!

@davidsandberg Could you provide some insight on how to freeze the trained models for deployment? How to strip off the training artifacts?

According to https://www.tensorflow.org/lite/tfmobile/prepare_models,

output_node_names is a list of the names of the nodes that you want to extract the results of your graph from. This is needed because the freezing process needs to understand which parts of the graph are actually needed, and which are artifacts of the training process, like summarization ops. Only ops that contribute to calculating the given output nodes will be kept. If you know how your graph is going to be used, these should just be the names of the nodes you pass into Session::Run() as your fetch targets.

The model seems cannot convert to CoreML directly, I have write a py script which convert to caffemodel first, and then use coremltools to convert to CoreML model.
You can refer to my git:
https://github.com/taylorlu/Facenet-Caffe
https://github.com/taylorlu/face_recognition_ios
import coremltools coreml_model = coremltools.converters.caffe.convert(('InceptionResnet_Model/inception_resnet_v1_conv1x1.caffemodel', 'resnetInception.prototxt2'), image_input_names='data', is_bgr=True) coreml_model.save('InceptionResnet_Model/InceptionResnet.mlmodel')


Incompatibility with Moode Audio Player

Hello, thank you for this awesome software! Unfortunately, it doesn’t work with Moode audio player which is a fork of Volumio. Any idea why? I’m using the current release of Moode, 7.6.1

If you could enable the debug log, try to connect to moode, and then attach the log here I might get an idea why. In any case the WAV header option won't work with Volumio or Moode.

Question: did you enable the upnp client for mpd in the UPnP/DLNA section (follows the Renderer section) in Moode?

Hello, upnp renderer is enabled I’ve attached two pictures of the failure screen I’m receiving in Moode.

One of the screens is with the wave header enabled and the other is not. With a lot of fiddling I was able to get white noise coming from the renderer, I used directsound instead of wasapi in Kodi, but that was not the sound it should have been pushing from Kodi in windows. I was trying to stream flac audio from a video file.

Sorry, the second image is incorrect, the second image should of mentioned a curl error. I’ll try to get you the debug logs and proper error message later today.

Hello the curl error was only with the wav header enabled. So I don’t think it’s relevant in this case, however the above photos are the error message without the wav header.

Without the debug log there's not much I can say.
But if Moode is anything like Volumio then

wav header won't work
24 bit will produce nothing but noise
chunked transfer is not recommended

Thank you for your assistance. With your settings I was able to get it running with Volumio, but there was a few limitations in my set up in comparison to tuneblade, similar software but airplay. I'll open a new and different issue and try to resolve this one if we can sort out the other one.

Hello, we just installed Moode Audio Player, we get the same error when trying to stream from SWYH-rs. I attached the debug log
log.txt
.

It looks like your Moode player is not configured for OpenHome Transport (the default for Volumio) but only for AV Transport, so swyh-rs falls back to using  AV Transport, but Moode responds with an internal server error 500 when setting the AV Transport URI :
13:54:48 [DEBUG] (1) ureq::unit: sending request POST http://<IP_ADDRESS>:49152/uuid-5e211097-4658-5161-a661-b827eb2065eb/ctl-urn-schemas-upnp-org-service-AVTransport-1
13:54:48 [DEBUG] (1) ureq::unit: writing prelude: POST /uuid-5e211097-4658-5161-a661-b827eb2065eb/ctl-urn-schemas-upnp-org-service-AVTransport-1 HTTP/1.1
Host: <IP_ADDRESS>:49152
Connection: close
User-Agent: swyh-rs-Rust/0.x
Accept: */*
SOAPAction: "urn:schemas-upnp-org:service:AVTransport:1#SetAVTransportURI"
Content-Type: text/xml; charset="utf-8"
Content-Length: 1049
13:54:48 [DEBUG] (1) ureq::unit: response 500 to POST http://<IP_ADDRESS>:49152/uuid-5e211097-4658-5161-a661-b827eb2065eb/ctl-urn-schemas-upnp-org-service-AVTransport-1
13:54:48 [ERROR] <= SOAP POST error: http://<IP_ADDRESS>:49152/uuid-5e211097-4658-5161-a661-b827eb2065eb/ctl-urn-schemas-upnp-org-service-AVTransport-1: status code 500

I Have not yet used Moode, but might switch to it and ditch Volumio (as Volumio 3 seems to have serious problems while Volumio 2 is rock solid).
Do you have the possibility to configure OpenHome transport in Moode? If swyh-rs finds both transports it will always use OpenHome, and if you're lucky it will work. Both Volumio and Moode use upmpdcli as an upnp/dlna front-end for mpd,
And perhaps Moode has an error log that you can check?

Would you be able to test with the latest 1.3.22 release? The DIDL-Lite for AV transport streamers was broken since 1.3.20 (it was OK for Openhome transport streamers).

I will test today or tomorrow.

I have now tested with Moode 8.02.
I can confirm that it works without any problem if:

you configure the UPNP client for MPD as Openhome renderer (in Edit UPNP Config)
and use the WAV header option.

The new 1.4 release works with Moode without problems with the new flac support.


differentiate between dependencies and dev-dependencies for the sake of size

Installing omnisharp-client consumes 96MB on disk while OmniSharp itself just the half (46MB). tsd and typescript, which seem like dev dependencies, sum up to 26MB. So some low hanging fruit here...

btw - I'd like to sign up for this.

Also it seems that RX can be slimmer. I see all files in dist/ being foo.[js|map|min.js]...

Sounds good to me!
I do know that as part of the install process we run the typescript task through the post-install and pre-publish commands for npm.  So we might have to fiddle with the build to get the proper JS output stashed to npm, without requiring the .ts files to be published (or publish them, but don't try to regenerate them).
Depend on what happens above we might also need to generate the declarations for local developers so they get the type information (today atom just references the ts files through the node_modules directory).
PS: Does this mean VSCode might be evaluating using the same dep as atom? :grinning:
If you want to try and pair sometime to figure what we need to do out, I can make time for sure.

Rx also has https://github.com/Reactive-Extensions/RxJS-Modular but it hasn't been updated in a few months, maybe @mattpodwysocki might have some input.
Worst case on the Rx front we could take a local dependency on it, and just copy the raw file from github into the release based on whatever release we're looking at, and then remove it as a direct dependency.

oh there's also https://github.com/Reactive-Extensions/RxJS-CLI maybe that could be useful.

I've got my changes @ https://github.com/OmniSharp/omnisharp-node-client/tree/diet-publish
@jrieken let me know if you want me to push them onto diet and we can merge it.

:+1:


[Python] Add support for Decimal32 and Decimal64

Describe the enhancement requested
Decimal32 and Decimal64 were added to the Arrow format in #43976. C++ implementations were added in #43957. Add support for these in PyArrow.
Component(s)
Python

When doing this, remember to add entries in arrays.rst and datatypes.rst.


Where should I put the config file for centos/rhel ?

I've searched around and it seems there is no straightforward answer to this.  I can't use the default version of the software and need to change many config options.  How can I do this?
I tried the solution found in #3285, but that didn't work for me.
I should add: when I run the -config command found in #3285, I get the following message:
[handoff] 2015/08/03 19:41;80 Failed to start hinted handoff processor: mkdir all: mkdir /root/.influxdb: permission denied

@ericjmiller please verify my understanding of the problem:

You downloaded InfluxDB 0.9.2 and installed the RPM.
You generated a new config file with influxd config > filename
You launched InfluxDB with influxd -config /path/to/filename and you immediately get the permissions error for /root/.influxdb?

Is that all correct?

Well, there was a lot of running the influxd config > filename in different directories, then restarting the server to see if the config file would work.  But yea, more or less that's what happened.
I'm just going to delete my ec2 instance and start over.

I figured it out after reading you steps and reinstalling.
For those that have the same problem in the future:
run this command in the directory you want to contain your config file:
influx config > filename
then enter the path to THAT file in the following command:
influxd -config path/to/file
By the way - I didn't see this info in the official docs.


force stop not working

before stop option isn't working. I have it set to do that after 5 minutes but when I go to the settings after the time the application is still running

https://github.com/brevent/Brevent/blob/38bcdbf7d31644360a6b9b7c416831b13ece2920/brevent/src/main/res/values/strings.xml#L22

This issue is closed because there is no log.


Qna penalty

issue #929 - penalize for answering too many questions

note this causes failed tests. Adds vuln and lic differently so they don't collide if more than one on blank board. I didn't fix tests - we can do later


[WIP] Insert and Remove Tracking

Copied from the items branch and finished it, thanks again for the help @Meronat
This currently adds

Logging insertion of items from tile entities
Logging removal of items from tile entities
Displaying of quantity for queries that have a quantity

This is pretty much my first time contributing something big like this and probably needs a bunch of testing before it should be merged, I'll deploy this later on my server with ~20 people online most of the time and see if we can break it.

Is it an eventual goal to allow the undoing of these actions?

I'm sure it is, but I think storing that extra data should be optional, since NBT on items can be quite long.
Anyway, during testing I ran into some issues with this, not all modded Inventories are an instanceof TileEntityCarrier so I'm currently trying to figure out how to recognize those and more importantly, getting the block position of the TileEntity they are part of.

I don't think I can figure this out myself, I'll close this for now.


mgmt, first beta for migrationdiscoverysap

Description
release request: https://github.com/Azure/sdk-release-request/issues/5009
namespace review:  https://github.com/Azure/azure-sdk-pr/issues/1737
specs PR: https://github.com/Azure/azure-rest-api-specs/pull/28130
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

/azp run prepare-pipelines

/azp run java - migrationdiscoverysap - ci

API change check
APIView has identified API level changes in this PR and created following API reviews.
com.azure.resourcemanager:azure-resourcemanager-migrationdiscoverysap


String marshalling fix

Hello again. I solved the issue I reported in #2. It was caused by native not being Utf8 on all platforms. I bumped CppSharp to the latest version and ran it on the header for Native File Dialog Extended v1.1.1.

Awesome! Gonna check that everything still works on my machine and stuff

If you know how to fix it, go right ahead. You should have push access to my branch.
Otherwise, I'll take a look tomorrow.

So far I've tried a somewhat naive approach of adding function name replacements (N to U8) in the PostProcess method, and I think it seemed to work up until it was time to find the nfdu8filteritem_t type. I think CppSharp can't find it yet again because it's a #define, but I can't find a function to rename that type...

I think the best solution might be to ask the native library author to make the API universal. A few small tweaks upstream and generating interop bindings becomes easier for all frameworks.

If we pr that change upstream, the utf8 methods will work out of the box on all 3 platforms.
However, the native methods will have cross platform differences. Currently, the CppSharp output code aligns with Windows wide characters. I propose we add SupportedOSPlatform attributes to the native methods. That would warn developers that they should only use the native methods on Windows.

https://github.com/btzy/nativefiledialog-extended/pull/118

https://github.com/AssetRipper/nativefiledialog-extended/actions/runs/7152679491

Well that was fast :v

I also want to pr doxygen support to the native library because that will let CppSharp pull the documentation for our library.

I also asked about injecting attributes.
https://github.com/mono/CppSharp/discussions/1804

I also asked about injecting attributes.
mono/CppSharp#1804

I was able to inject the attributes!

A few updates:

My upstream changes were merged!
I set up automated building. https://github.com/AssetRipper/nativefiledialog-extended-builds/actions

Oh my god... I didn't have any Readme file all this time?? This is bad :skull:

I'll update NativeFileDialogs.Net.Runtime now

Much appreciated

If you know how to fix it, go right ahead. You should have push access to my branch.
Otherwise, I'll take a look tomorrow.

Well... apparently I don't! D:
❯ git push AssetRipper HEAD:string-marshalling-fix
ERROR: Permission to AssetRipper/NativeFileDialogs.NET.git denied to Speykious.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

What were you trying to change? Just bumping the NuGet version?

Yes, on all packages for NativeFileDialogs.Runtime, and also change the submodule since we don't need my fork anymore.

I don't think you can push to my branch, probably because I made this from an organization repository.

I see. Is there anything you can do in the organization itself then?

I added permissions for you to push to that repository, so you should be good now.

Thanks, it worked :D

The examples work on my end with the new versions! I'll do one more test before merging.

I noticed that you were referencing the Autogen NuGet package. It's only necessary to reference the Runtime NuGet package due to how native libraries are used.

Oh damn, it has actually never worked on Mac. I'm getting a DllNotFoundException. It can't find the libnfd.dylib in the right place no matter what name I give the runtimes/osx-... directory. I guess that's a separate issue I'll have to pull up.

I noticed that you were referencing the Autogen NuGet package. It's only necessary to reference the Runtime NuGet package due to how native libraries are used.

What do you mean? As in I should turn AutoGen into a project reference instead?

https://github.com/Speykious/NativeFileDialogs.NET/pull/3/commits/613358ec52f5ddebaf6474fcd3a7c20518c3f530

Whenever you compile, all the NuGet references will be correct in the generated packages.

Ah right. Yeah that's what you meant. That's fine


Base64 encode WOFF FuelUX font to reduce requests

Convert WOFF Fuel UX font to base64 and store in CSS file.
For more information on fonts visit: https://css-tricks.com/snippets/css/using-font-face/
@futuremint We can do the same with BS3's glyphicon font, but that will probably be a rucksack issue @kevinparkerson.

Follow up: From what I can ascertain from Chrome Timeline. The base64 decode may be adding 10-20ms to rendering time (It's within multiple reload margin of errors though).
Right now, network download from Akaimai of WOFF file ranges from 225ms to 900ms (although I am on a plane right now with pretty bad latency).
HTTPS requests from Akaimai are taking 500ms to 1.5s (but then again, I'm on a plane).
We probably shouldn't be encoding large fonts, but this small one seems like a good idea for optimization.


Settings checkboxes are no longer clickable

Description of Issue
Somehow the checkboxes in the settings are no longer clickable.
The last things I did was:

Activate the debug options in the plugin and activate the -D switch in homebridge settings.
Deactivate -D switch in homebridge settings.
Update the plugin from 2.0.0 to 2.1.0
Unchecking another yet activated checkbox works, but it is not possible to check it again afterwards.

Tested with FireFox 132.0b4 & Safari 18.0.1
Even the downgrade to plugin version 2.0.0 does not make the checkboxes clickable.
Plugin Version
2.1.0
AEG/Electrolux Appliance(s)
No response
HomeKit App(s)
No response
Diagnostic Checks

[ ] AEG iPhone app is able to control the appliance.

Log File
No response

They work fine for me in:

Chrome Version 129.0.6668.90 (Official Build) (64-bit) on Windows 11.
Safari on iOS 18.0.1.

Do checkboxes work in the configurations for other plugins?
The configuration is just provided as a JSON schema. It is Homebridge UI that renders it into a webpage. If there is an issue then it will be with that plugin, not this one.

I just updated homebridge from 1.3.6 to 1.3.7 and then it worked just fine.
After that I got noticed about homebridge-config-ui-x update from 4.59.0 to 4.60.1 again.
And that seems to be responsible for the problem.

Also, you say that the problem persists with older versions of this plugin which previously worked for you.

I think this was before updating to homebridge-config-ui-x 4.60.1
I will try to uninstall the plugin and delete the config and then reinstall.

Having just cleared my browser crashes I can now see the issue with the checkboxes in Homebridge UI. It appears to affect all plugins that use a static config.schema.json, but not those that use a custom UI via homebridge/plugin-ui-utils. Hence, this is definitely a problem with homebridge-config-ui-x rather than this plugin.

Ah great! Can you (as a plugin developer) report this issue to homebridge-config-ui-x?

This is fixed in homebridge-config-ui-x 4.60.2-beta.1.
See homebridge/homebridge-config-ui-x#2200 and homebridge/homebridge-config-ui-x@09be61beb0d142836cfecf488b266538213f1ea3.

Cool, thanks for the info!


Add GH action to update middleware READMEs

Docs for the middleware modules linked from https://expressjs.com/en/resources/middleware.html are sourced from
the README files in the corresponding repos and are copied to _includes/readmes using get-readmes.sh. So, e.g., https://expressjs.com/en/resources/middleware/body-parser.html comes from https://github.com/expressjs/body-parser/blob/master/README.md.
Similar to #1504 this issue is a task to add a GH action to perform this task.  However, since it should run whenever any of the middleware READMEs are updated, it might require an action be added to each of those repos so that it would get triggered when that occurs....unless there's a way to detect a change to a particular file in another repo and perform an action when that happens. Since I am not an expert on GH actions, I will defer to someone who is.
Also some community docs from the expressjs/express repo are similarly reproduced on the website in Resources > Contributing to Express, cf. https://github.com/expressjs/expressjs.com/pull/1507 and it would be great to automate that, too.

I have no experience with this kind of thing but I am available to give this one a shot. Unless it requires admin GH permissions?

Awesome!
Re admin rights, I'm not sure tbh.  Maybe @jonchurch knows.

#1559

@crandmck The workflow could not be executed because the event peter-evans/create-pull-request@v5 is not allowed in the repository. If allowed, we could verify if the workflow works correctly. Alternatively, I can modify the workflow to use gr2m/create-or-update-pull-request-action@1, which is allowed in the repository.

Fixed by https://github.com/expressjs/expressjs.com/pull/1559


Handling saga IO from within callbacks

Yielding a call to a function that is using a callback to run its post execution behavior: yield apply(someObject, someFunction)
Seems I can't yield put from someFunction since the callback isn't a generator. How can I dispatch an action from that callback?

There is a cps effect that works with node style callbacks (err, res). But for now only works with non method functions (or strongly bound functions) [I'll have to fix that]
const result = yield cps(func)
yield put(myAction(result))

Another solution is to promisify your method and use it with call instead of cps
function promisifiedFn(...args) {
  return new Promise( (resolve, reject) => {
    obj.method(...args,  (err, res) => err ? reject(err) : resolve(res)
  }
}

Thank you! Looking forward to the apply version of cps.
Is it okay to listen for DOM events like this?
const waitForEvent = (el, type) => new Promise(resolve => el.addEventListener(type, event => resolve(event)))

function* listenToClick() {
  while (true) {
    const el = document.querySelector('.some-class')

    const event = yield call(waitForEvent, 'click')
    yield put(someActionCreator(event))
  }
}

Just noticed 0.5.0 is out with improved cps and it works fine for this issue's original use case, but I haven't managed to use it to wait for a DOM event, I get listenToClick: uncaught MouseEvent:
function* listenToClick() {
  while (true) {
    const el = document.querySelector('.some-class')

    const event = yield cps([el, el.addEventListener], 'click')
    yield put(someActionCreator(event))
  }
}

addEventListener calls its callback with the event as a first parameter. cps's callback is called with the event as a first argument by addEventListener, since cps expects an error first argument. it treats the event as an error.
I'd opt for your first solution but would remove the event listener after the promise is resolved
const waitForEvent = (elm, eventType) => {
  return new Promise(resolve => {
    const cb = e => {
      resolve(e)
      elm.removeEventListener(eventType, cb)
    }
    elm.addEventListener(eventType, cb)
  })
}

Alright, thanks for the tips :+1:

Hi @yelouafi !
Thanks for your awesome work, but I have a question here: how to remove listeners in the automatically canceled race effects?


flaw in hashmap_get

#include <assert.h>
#include <stdio.h>
#include "hashmap.h"

int main() {
    struct hashmap_s m;
    const char* key = "<IP_ADDRESS>hv_api.udache.com/abc/def";
    int len = strlen(key);

    char* value = "1";
    void* ptr = NULL;

    hashmap_create(1024, &m);
    hashmap_put(&m, key, len, value);

    ptr = hashmap_get(&m, key, len);

    /* should not get NULL */
    assert(ptr != NULL);
    return 0;
}

Reason analysis:

In hashmap_hash_helper which is called by hashmap_put, the variable curr was increased by 8 times.
But in hashmap_get,  Although curr was increased by 8 times,  hashmap_match_helper was executed only 7 times.


Custom HTTP status codes for DDS checker

Hi Holger!
First of all, thank you for this fantastic piece of software!
I'd love for DDS checkers to support custom HTTP status codes. For example, the website that I'm scraping is returning 302 codes for items that no longer exist. That would also mean checking for xpath values on redirected urls.
Looking forward to hearing your thoughts on that.

Hi, would be open for that, though I'm not having that much time atm to do the changes on short term (also sorry for the late answer).
After reflecting on this a couple of minutes, I think, following might be a good implementation:

Replacing 404 type with CUSTOM_HTTP_CODE type
Replacing 404_OR_X_PATH type with CUSTOM_HTTP_CODE_OR_XPATH type
Adding a field checker_custom_http_code which defaults to 404

Like this backwards compatibility would also be kept (existing settings wouldn't break).
What do you think? Does this makes sense?

Thanks for the answer. Don't worry about short term changes, I'd rather have them solid instead of rushed.
If you're worried about backwards compatibility, maybe adding a CUSTOM_HTTP_CODE instead of replacing the 404 type makes more sense. Otherwise I think your suggestions sound perfect.


Restrict file system access

Currently Money have read/write access to then user's entire home directory, this leads GNOME Software to flag it as "unsafe".

Flatpak have the concept of "Portals" which GTK implements:
https://docs.flatpak.org/en/latest/portals.html#portal-support-in-gtk
This grants the application access to files picked from the GTK file picker dialogs, so I think that Money doesn't really need "--filesystem=home". An alternative would be to scope it to something like the Documents subdirectory, so it doesn't have access to all the files in the user's home directory.
https://docs.flatpak.org/en/latest/sandbox-permissions-reference.html#filesystem-permissions
Example: "--filesystem=xdg-documents".

Permission access has always been a problem with C++ libraries. Although permission is granted from the portal on flatpak, C++ libraries (such as sqlite) still give me permission access denied when trying to open a file provided by the path from the portal. Hence why permission access is set to home. See #107 . Even though the user created the file from the portal and opened it. The app still couldn't access the file. Now, this occurred on a NAS, but the same issue happens if home access is removed and a user tries to create a file in the home directory.
So for now, permissions will stay as they are.

I was thinking about this problem today, and I have one solution. I first rejected it because it's a big change, but since we're rewriting the app, maybe you will accept this, @nlogozzo. But this is a HUGE UX change IMO.
There's the CSV export option. And it's not like *.nmoney files can be used somewhere else really. So what's reason why we let the user choose where to store these files? Except for syncing purposes ofc.
So why not to store files in the app's data folder (~/.var/app/ID/data for flatpak or ~/.local/share/app)? This way we will not need any filesystem access. We can also easily list all accounts on start screen (and we'll have to - because there will not be "Open" button), sorted in the order of last used.
And for syncing, well, the user will not have the ability to place an account file anywhere, but all accounts will be in one place, inside home folder (yes, deep inside, but we can provide some hints in the app how to find the folder), so I don't think it would be much of a problem to configure syncing app to grab files from here. Or a systemd service to copy files from account folder to somewhere else can be easily written (I can do this if needed).

@nlogozzo Well it could be changed from "--filesystem=home" to "--filesystem=xdg-documents" that way the application would have access to a more limited scope. That should increase safety and trust. perhaps that would eliminate the "Money is unsafe" warning in GNOME Software.
@fsobolev I think a systemd service sounds rather undesirable.

And for syncing, well, the user will not have the ability to place an account file anywhere, but all accounts will be in one place, inside home folder (yes, deep inside, but we can provide some hints in the app how to find the folder), so I don't think it would be much of a problem to configure syncing app to grab files from here. Or a systemd service to copy files from account folder to somewhere else can be easily written (I can do this if needed).

This could be solved by providing a backup location option where if set the app could copy the nmoney file to the destination. A warning about lacking permissions could be provided here and the user can manually add the required permission using flatseal.

If we're going to go to all this trouble for a systemd service or a backup location the app will still need permission to the home directory to be able to be backed up anywhere. Which at that point we might as well leave it the way it is.

If we're going to go to all this trouble for a systemd service or a backup location the app will still need permission to the home directory to be able to be backed up anywhere. Which at that point we might as well leave it the way it is.

My point was that backing up these files aren't something everyone would do and for the people who would want to do it, it wouldn't be too hard for them to figure out how to give access to the home directory using something like flatseal. Alternatively we could provide a button for the user to view the folder where the files are stored and from there they could use their own backup solutions like @fsobolev mentioned.


less copying, more moving! :truck:

Scope & Purpose
Copy less data, but move more :truck: :truck: :truck:

[x] Bug-Fix for a released version (did you remember to port this to all relevant release branches?)
[ ] The behavior in this PR can be (and was) manually tested (support / qa / customers can test it)

Testing & Verification
This change is already covered by existing tests, such as all cluster tests.
https://jenkins01.arangodb.biz/view/PR/job/arangodb-matrix-pr/5575/

Tests blue


Record timestamp for when posts/comments were saved

The new saved posts/comments works nicely! It might be beneficial to gather the timestamp of when a post/comment was saved, so a user can sort by that, instead of sorting by the creation date of posts/comments

Oh, do you mean like, when you hit the "save" button on a given comment vs when the comment was made?
I don't remember if that info is included anywhere, but I can check! that would be useful.

Yup, that's what I mean! Peeking at the GDPR archive, it looks like it's not included, and the saved comments/posts don't seem to be ordered that way either.
It may not be possible to get the saved date, but the order of saving should be possible to retrieve from the API, as that's how reddit displays it through the website

cool! i'll take a look

So this one is tricky- the tool currently only loads "saved" data from the archive, which (as far as I can tell) is unordered. I go and fetch info about the data, but that doesn't know anything about the user that saved it (since I just look it up by id).
There's a private (unauthenticated) feed that includes all the data and the ordering, but I'd need to add a new command to cover that and I don't know that I'll have time to write and test that this week (before the API changes go into affect saturday). I can get it afterwards, but I don't want to make any promises before we know more about the nature of the changes.
As a stop gap, here's a little script to get the absolute ordering of the data right now and we can get it into the sqlite database at our leisure:
https://gist.github.com/xavdid/f0999e3ea08cc8cdaafce27618e092fd
Download the script, make sure requests is available (pip install requests), and run python saved_ordering.py > ordering.json
I might be able to get the real feature shipped in time, but this should at least preserve your data in case I don't.
Let me know if you need further instructions with the above, too! I'm presuming a bit of Python knowledge.


Reduce size of Decimal type

The Decimal type is 48 bytes on a 64-bit machine. Even though this is not "large" per se, it seems likely that users would construct large slices of Decimal values, so it's worth trying to reduce the size to pack more values into less memory. There are two relatively easy ways to do this:
First, you can reorder the Decimal fields from this:
type Decimal struct {
	neg bool
	coef bint
	prec uint8 // 0 <= prec <= 19
}

to this:
type Decimal struct {
	coef bint
	neg bool
	prec uint8 // 0 <= prec <= 19
}

Putting neg and prec together after the coef reduces the padding used for alignment, saving 8 bytes.
Second, unless I'm missing something, you could refactor bint to remove the overflow bool. It looks like you could implicitly infer that overflow == (bigint != nil). This saves an additional 8 bytes, since Go pads the boolean to make bint a multiple of 8 bytes. Unlike Decimal, in this case there's no way to reorder the fields to reduce the size.
(You could alternately reduce the size by removing bint, rolling the fields into Decimal, and placing overflow alongside neg and prec, but doing this loses the encapsulation of bint and requires a larger refactor than just replacing all instances of overflow with bint != nil.)

The actual implementation of Decimal type is already your suggestion, I'll update the README to match it. As for overflow, agree that it can be removed and checked by bigint != nil instead. I'll craft a fix for it. Thanks


When .inObjectScope instance is resolved inside .initCompleted, there are temporarily two instances created

import Swinject

let container = Container()

protocol FooProtocol {}

protocol BarProtocol {}

class Foo: FooProtocol {
    var bar: BarProtocol?
}

class Bar: BarProtocol {
    let foo: FooProtocol

    init(foo: FooProtocol) {
        self.foo = foo
    }

    deinit {
        print("This print shouldn't happen")
    }
}

container.register(FooProtocol.self) { resolver in
    return Foo()
    }.inObjectScope(.container).initCompleted { resolver, foo in
        let foo = foo as! Foo
        foo.bar = resolver.resolve(BarProtocol.self) as! Bar
}

container.register(BarProtocol.self) { resolver in
    let foo =  resolver.resolve(FooProtocol.self)!
    let bar = Bar(foo: foo)
    return bar
    }.inObjectScope(.container)

let myBar = container.resolve(BarProtocol.self)!

This code will create two Bar instances and instantly deinit one of them. Shouldn't it check if there ~is~ will be an actual instance inObjectScope ~already~ created? Not sure if this problem is solvable.

You create a first Bar (first Bar doesn't exist)
inside it's init you make first Foo (first Bar doesn't exist, first Foo doesn't exist)
First Foo completes initialization and triggers initCompleted (first Bar doesn't exist, first Foo exist)
Inside initCompleted it created second Bar (first Bar doesn't exist, first Foo exist, second Bar doesn't exist)
Second Bar injects first Foo as it exists already (first Bar doesn't exist, first Foo exist, second Bar doesn't exist)
End of second Bar's init. Second Bar created (first Bar doesn't exist, first Foo exist, second Bar does exist)
End of initCompleted for first Foo (first Bar doesn't exist, first Foo exist, second Bar does exist)
End of first Bar's init (first Bar does exist, first Foo exist, second Bar does exist)

So my question is: myBar gets the reference to which one? 1st or 2nd Bar?

We are aware of the behavior (see docs). We have tried to solve this issue (#134) but it proved to be a lot more complicated, so we ended up reverting the change (#164) - you can find discussion & reasoning in #159

I didn't looked at the "Remark" section in documentation, sorry! Thanks for taking your time to answer in such a detailed manner :)
Anyway which instance is injected? The "inner" or "outer" one?
I think the issue can be closed :)

If I remember correctly, it should be the "outer" one, but I wouldn't take my word for it 😄 Should be easy enough to check though.


[SPARK-44194][PYTHON][CORE] Add JobTag APIs to PySpark SparkContext

What changes were proposed in this pull request?
This PR proposes to add:

SparkContext.setInterruptOnCancel(interruptOnCancel: Boolean): Unit
SparkContext.addJobTag(tag: String): Unit
SparkContext.removeJobTag(tag: String): Unit
SparkContext.getJobTags(): Set[String]
SparkContext.clearJobTags(): Unit
SparkContext.cancelJobsWithTag(tag: String): Unit

into PySpark.
See also SPARK-43952.
Why are the changes needed?
For PySpark users, and feature parity.
Does this PR introduce any user-facing change?
Yes, it adds new API in PySpark.
How was this patch tested?
Unittests were added.

cc @juliuszsompolski @WeichenXu123 @hvanhovell

Merged to master.


Selling to traders breaks the game when reaching a certain player level or time played.

General information
Project issue is for:
3.8.0 - BLEEDINGEDGE (28476)
Ive ran two new accounts with 3.8 to around level 40 with zero mods. First was zero to hero and second was EOD version.  When I get to around level 40 my inventory and traders bug out (I get the spinning wheel in the traders inventory window and can only play if I dont buy or sell anything to traders). I deleted the first account and im on the second one now and it has the traders bug Im referencing.
How to reproduce
Its hard to say whats causing it im not sure how it works. It could be the specific level Im making it to or it could be the amount of time i put into the account. Both accounts had around 40 hours when the bug happened.
Log files
Server: Attached
BepinEx: attached
Client: attached
Profile
I believe running my profile will show the bug. It only happens when I sell to the traders. I can do anything else if I dont try to sell anything but if i sell something and try to enter a raid It wont let me enter.

Attachments

server-2024-03-07.log (94 KiB)
LogOutput.log (51 KiB)
2024.03.07_12-56-16_<IP_ADDRESS>.28965 traces.log (88 KiB)

Originally written by ledski

Zip up your current profile and attach it to the issue please, otherwise could be significantly harder to track down

Originally written by Cj

Profile

Attachments

65ccc87b0000af08ac2c4c2e.zip (132 KiB)

Originally written by DrakiaXYZ

Fence is sending partial stacks of money... But this shouldn't be possible with the client calculated Sell All value
{
	"_id": "b70d28bd5c8f76b19a2c49b5",
	"uid": "579dc571d53a0658a154fbec",
	"type": 13,
	"dt":<PHONE_NUMBER>,
	"text": "",
	"templateId": "64648f64063abf54100f587b 0",
	"hasRewards": true,
	"rewardCollected": false,
	"items": {
		"stash": "d8d8a7a6c193e8738dda6d27",
		"data": [
			{
				"_id": "f6c60e0fde75d3829007879a",
				"_tpl": "5449016a4bdc2d6f028b456f",
				"upd": {
					"StackObjectsCount": 224980.5
				},
				"parentId": "d8d8a7a6c193e8738dda6d27",
				"slotId": "main"
			}
		]
	},
	"maxStorageTime": 259200
},

Potentially running an outdated server executable? This shouldn't be possible with fully up to date code

Originally written by DrakiaXYZ

Confirmed with user that the profile is from a build prior to the Fence sell all fix, so this isn't an actual issue. Closing

Originally written by DrakiaXYZ


Fix quickstart link in /README.md

Motivation
In main README, the link to the quickstart guide goes to the upstream instructions that does not work for ODH:
Modifications
The link is fixed to point to the right quickstart guide.

Ok my bad, I did not noticed that my approval was enough to automerge. @israel-hdez Please implement JooHo comment in another PR

Ok my bad, I did not noticed that my approval was enough to automerge. @israel-hdez Please implement JooHo comment in another PR

Too much automation :-)
@danielezonca, @Jooho Here is it: #109


Update build.gradle

It looks like the version of awaitility in this gradle configuration file is not correct.

@MaoPingZou Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

I'll leave this open as a reminder, because it's correct. However, this guide is about to be re-written, and we'll catch this and other problems when we do that. For now, I don't recommend trying to fix it.

Same comment as on https://github.com/spring-guides/gs-scheduling-tasks/pull/26, but I'll close this one, since we need only reminder about the awaitility version.


Ability to use IAM Role with AWS S3

Is your feature request related to a problem? Please describe.
Currently when using TS and AWS S3 it requires IAM User secrets to be able to send data to AWS S3. I'd like to be able to use the IAM Role assigned to the BIG-IP to send the data to S3.
Describe the solution you'd like
The ability to use IAM Role, with appropriate AWS S3 permissions, assigned to the BIG-IP as the authentication/authorization to AWS S3.
Describe alternatives you've considered
The internal "Cloud" team doesn't allow IAM User secrets in AWS, the requirement is to use Roles assigned to the EC2/BIG-IP instead of Users.

Thank you for your feedback, this is being tracked in our internal product backlog as AUTOTOOL-1340.

This issue has been resolved as part of the 1.12.0 release.


No such file or directory: u'data_reader/feats/la_dev_spec_tensor4.scp'

Hi @jefflai108 ,
What are the steps to get this file?
I am running:
$ python main.py
INFO - asvspoof19 - Running command 'main'
INFO - asvspoof19 - Started run with ID "185"
squeeze-and-excitation network
===> Model total parameter: 1343762
===> loading checkpoint 'snapshots/181/model_best.pth.tar'
===> loaded checkpoint 'snapshots/181/model_best.pth.tar' (epoch 6)
ERROR - asvspoof19 - Failed after 0:00:02!
  File "main.py", line 675, in main
    forward_pass()
  File "main.py", line 302, in forward_pass
    val_data    = SpoofDatsetSystemID(data_files['dev_scp'], data_files['dev_utt2index'], binary_class=True)
  File "/home/infcodey/repos/ASSERT/assert/data_reader/dataset_v1.py", line 154, in __init__
    with open(scp_file) as f:
Traceback (most recent calls WITHOUT Sacred internals):IOError: [Errno 2] No such file or directory: u'data_reader/feats/la_dev_spec_tensor4.scp'

I'm trying to run the binary SENet34 model (best performing one) on the dev & eval set.
Thank you

features/run_feature.sh


Feature Request: Switch to Disable Query field getting truncated to 1024 characters

The Query column in the Vizql_End_Query table and the Query_Abstract column and Query_Compiled columns in the vizql_qp_batch_summary_job table are getting truncated to 1024 characters.
We'd like the complete query to be logged as a partial query doesn't have much utility. An option to disable the truncation or set a new limit would be wonderful.
Thank you!
Ray
    public VizqlEndQuery(BsonDocument document)
    {
        ValidateArguments("end-query", document);

        SetEventMetadata(document);
        BsonDocument values = BsonDocumentHelper.GetValuesStruct(document);

        Query = BsonDocumentHelper.TruncateString(BsonDocumentHelper.GetString("query", values), 1024);
        ProtocolId = BsonDocumentHelper.GetNullableLong("protocol-id", values);
        Cols = BsonDocumentHelper.GetInt("cols", values);
        Rows = BsonDocumentHelper.GetInt("rows", values);
        QueryHash = BsonDocumentHelper.GetNullableLong("query-hash", values);
        Elapsed = BsonDocumentHelper.GetDouble("elapsed", values);
    }

Thanks Ray!  I believe this truncation was to avoid performance issues on logsets with extremely large queries (see MDX queries).  We'll likely address this by including a plugin argument flag to override the default behavior with a custom limit.

This has been mitigated somewhat by increasing the limit to 10000 characters in Logshark v2.0.


Submission status fix

Description of the PR
Rating of not yet graded submissions should show "In grading". As assignements which are submittable from the IDE are already past their deadlines I can't test it properly. I did as I @nikke234 advised me to so I hope everything is ok. Let me know if there are any comments from you.
Testing

unit
integration
manual

- [ ] new unit tests created
- [ ] all unit tests pass

- [ ] new integration tests created
- [ ] all integration tests pass

- [ ] manual testing went well

Have you updated the TESTING.md or other relevant documentation?

[ ] Yes
[ ] Not yet. I will do it next.
[x] Not relevant

Hi @StanislavFranko and thanks for opening a pull request!
Currently, this PR seems to be missing your own commits. Did you remember to push them?
Once you have pushed your commits, you can also change the PR's base branch to test (that is our "dev branch"; you can change the base branch by clicking Edit button next to the PR title). Or, I can do that too, but only after you have pushed your commits to this PR.
Looking forward to see your code! 😄

Yeah, and one more thing: If you like, you can append your GitHub username in the "Credits" part of README.md. :)

I had to close and re-open this to re-trigger Travis build.

Can you make the unit test comply with the changes?
fi.aalto.cs.apluscourses.presentation.exercise.SubmissionResultViewModelTest > testSubmissionResultViewModel FAILED

    org.junit.ComparisonFailure at SubmissionResultViewModelTest.java:20


[WIP] New gadget netcost exporting metrics on Prometheus

New gadget netcost exporting metrics on Prometheus
This new gadget counts bytes and packets sent and received in all pods and expose the metrics in the Prometheus format. It's possible to give a list of CIDRs that will annotate the metrics, so that we could distinguish in-cluster traffic and outside traffic. This is useful because the price of network traffic can vary depending on where the traffic is going to.
How to use
Example with kubectl-gadget: TODO.
Example with local-gadget:
$ sudo ./local-gadget
» create netcost trace1
State: Started
» operation trace1 start
State: Started
»

Generate some traffic
Check the counters:

$ curl -s http://localhost:19100/metrics|grep ^netcost
netcost_bytes{cidr="<IP_ADDRESS>/0",direction="egress",pod="default/loving_ritchie"} 4430
netcost_bytes{cidr="<IP_ADDRESS>/0",direction="egress",pod="default/minikube"} 176018
netcost_bytes{cidr="<IP_ADDRESS>/0",direction="ingress",pod="default/loving_ritchie"} 91691
netcost_bytes{cidr="<IP_ADDRESS>/0",direction="ingress",pod="default/minikube"} 165025
netcost_bytes{cidr="<IP_ADDRESS>/32",direction="egress",pod="default/loving_ritchie"} 3908
netcost_bytes{cidr="<IP_ADDRESS>/32",direction="ingress",pod="default/loving_ritchie"} 64045
netcost_bytes{cidr="<IP_ADDRESS>/8",direction="ingress",pod="default/minikube"} 11111
netcost_packets{cidr="<IP_ADDRESS>/0",direction="egress",pod="default/loving_ritchie"} 59
netcost_packets{cidr="<IP_ADDRESS>/0",direction="egress",pod="default/minikube"} 1113
netcost_packets{cidr="<IP_ADDRESS>/0",direction="ingress",pod="default/loving_ritchie"} 57
netcost_packets{cidr="<IP_ADDRESS>/0",direction="ingress",pod="default/minikube"} 1078
netcost_packets{cidr="<IP_ADDRESS>/32",direction="egress",pod="default/loving_ritchie"} 60
netcost_packets{cidr="<IP_ADDRESS>/32",direction="ingress",pod="default/loving_ritchie"} 60
netcost_packets{cidr="<IP_ADDRESS>/8",direction="ingress",pod="default/minikube"} 35

Testing done
[Describe the testing you have done before submitting this PR. Please include both the commands you issued as well as the output you got.]

@alban do you have any plans to move this forward?

This PR should be rewritten:

It could be both a "top" gadget (reusing the columns package) and a Prometheus gadget (reusing @flyth 's Prometheus work in #1384)
Reusing pkg/gadgets/internal/networktracer like other networking gadgets
The CIDR should be automatically selected with the Kubernetes Pod CIDR, Kubernetes Services CIDR, and other CIDR.
Rebased on the "gadgets self-contained" PR


[#noissue] Cleanup pom.xml

Codecov Report

Merging #6382 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #6382      +/-   ##
==========================================
+ Coverage   39.62%   39.63%   +<.01%
==========================================
  Files        3065     3065
  Lines       91444    91444
  Branches    11842    11842
==========================================
+ Hits        36239    36240       +1
  Misses      52215    52215
+ Partials     2990     2989       -1

Impacted Files
Coverage Δ

.../hbase/schema/reader/xml/XmlHbaseSchemaParser.java
50% <0%> (-3.85%)
:arrow_down:

...rc/main/java/com/navercorp/pinpoint/test/Item.java
66.66% <0%> (-3.34%)
:arrow_down:

...navercorp/pinpoint/rpc/common/SocketStateCode.java
82.92% <0%> (-2.44%)
:arrow_down:

...cluster/zookeeper/ZookeeperClusterDataManager.java
54.83% <0%> (+1.93%)
:arrow_up:

...pinpoint/profiler/instrument/AccessorAnalyzer.java
65.71% <0%> (+8.57%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e39945b...c4e70d7. Read the comment docs.


New rule: Build actions should have a single task

The nodes <Content>, <Compile>, <None>, <EmbeddedResource>, and <AdditionalFiles> can use ;'s to define multiple includes/excludes/removes/updates at once. This makes things harder to read, as lines can become long.
However, should we exclude the following case? <EmbeddedResource Include="*.txt;*.xml" />, so if the task defines an extension?!

If possible I think allowing a task to define multiple extensions is fine. But I don't think it's very bad to enforce them to be split up.


Question: Does the new skipCopySameType improves the performance?

I wanted to know what is the performance impact of that esp in terms of cpu / latency.

Likely yes. Goverter will generate less code when this setting is enabled.

Thanks!


timestamp interpretation problem with find 4.4.2

The bootstrap code for straight.el fails for me.
The download is o.k.
But the 2nd invocation fails:
straight--cache-package-modifications: Command failed: find . -name .git -prune -o -path ./straight.el/\* -newermt 2017-08-10T13\:21\:37\+0200 -print:
/usr/bin/find: I cannot figure out how to interpret ‘2017-08-10T13:21:37+0200’ as a date or time

I tried this on Ubuntu 14.04 (GNU findutils) 4.4.2
It works on Ubuntu 16.04:
find (GNU findutils) 4.7.0-git

From a users point of view it is important that the package installer "just works".
I solved my problem by installing a newer find-utils on my machine.
You could make the "%FT%T%z" customizable.
That way it is possible to fix the problem without a need to edit the source code or to install a newer find-utils package.

I agree that the find interop should be made more customizable. However, I would also like to set the defaults such that they work on as many machines as possible. This is the only way for the package installer to "just work".

One way would be to run "find --version" and extract the version number. Use (version< ...) to set the time-stamp-format-string. However this will break when a user updates find-utils. No idea what version of find-utils is used on the also mentioned BSD systems...

Nah, checking the version should be fine, as long as it is fast enough to be run on every startup. The real question is how to handle the old version: must the time zone be dropped, or is there an alternative syntax that can be used?

Well, OK, checking the version is more complicated than I thought, because BSD find does not have any --version flag.

FYI, I'm holding off on making the time format customizable because my current to-do item is fixing #41, which requires doing some refactoring of this part of the codebase. But I would not object to merging a pull request for that, in the meantime.

This should be fixed by https://github.com/raxod502/straight.el/commit/e7b096d13e261a17c66f69932009b71e2295dc57, please test.


fix: DocsCategory should support dir index page

If the page is an index page under the folder of the next level, it should be counted.

I'm actually considering an index-level configuration to display a deeper level of files on the current index page. What do you think?

Make sure to format the code with prettier

Thanks for contributing


ubuntu

Environment
Knex version:0.14.6
Database + version:mysql
OS:ubuntu

Select applicable tempalate from below.
If issue is about oracledb support tag @ atiertant. For MSSql tag @ smorey2 .
Rest of dialects doesn't need tags.
Bug

Explain what kind of behaviour you are getting and how you think it should do

Error message

Reduced test code, for example in https://npm.runkit.com/knex or if it needs real
database connection to mysql or postgresql, then single file example which initializes
needed data and demonstrates the problem.

Feature discussion / request

Explain what is your use case

Explain what kind of feature would support this

Give some API proposal, how the feature should work

Missing / erroneus documentation
Send issue to documentation repo, or fix it and send PR https://github.com/knex/documentation
Questions about how to use knex
Github issues are for knex development. Please send questions how to use knex to
stackoverflow or ask about it in gitter chat.

Please don't just copy paste unchanged template, write a proper report.

And by the looks of a screenshot that you have posted, this has nothing to do with knex.

Probably just missing the npm install step, or that particular project doesn't even have mysql listed as dependency in package.json.


Crategen remove hyper dep

Just fixing up a small thing I noticed, where we're relying on the Debug impl of hyper::StatusCode to produce correct status code checks, which feels very brittle.
Changed this to use StatusCode::as_u16 instead for comparison with the botocore-provided status code value.
As a nice side-effect, we can drop the hyper dependency from service_crategen.

@matthewkmayer this should be ready for your feedback!

I agree it appears a bit brittle to rely on hyper's Debug behavior for codegen. The change was made to use hyper's types instead of ints for stronger typing. See https://github.com/rusoto/rusoto/issues/522 and https://github.com/rusoto/rusoto/pull/523 for when this change happened.
I'm not sure if the tradeoff of simpler, potentially more robust codegen is worth the cost of less strong typing of generated code.
Perhaps it's an okay trade to make since service definitions have entries like "httpStatusCode":400.

Ah, thanks for that history lesson! It’s still not quite clear to me
what types of bugs this aims to prevent, but could the same (generating
code that compares enum variants) be perhaps achieved while still
avoiding relying on Debug?
I‘m thinking if crategen had a map from status code to enum variant
name, we could be generating the same code as before, without the
dependency on Debug, at the cost of maintaining a correct mapping?
On Thu, Mar 8, 2018, at 4:56 PM, Matthew Mayer wrote:

I agree it appears a bit brittle to rely on hyper's Debug behavior> for codegen. The change was made to use hyper's types instead of
ints> for stronger typing. See
https://github.com/rusoto/rusoto/issues/522 and> https://github.com/rusoto/rusoto/pull/523 for when this change
happened.>
I'm not sure if the tradeoff of simpler, potentially more
robust codegen> is worth the cost of less strong typing of generated code.
Perhaps it's an okay trade to make since service definitions have
entries like "httpStatusCode":400.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/rusoto/rusoto/pull/995#issuecomment-371387767

I think it was aimed less at bug prevention and more at code readability.
Reviewing this change again, another point for accepting the PR: less places to keep track of dependencies as codegen's version of hyper is different than the actual code's version, which could cause bugs.
How about leaving the PR open for a few days to think about it and see if someone else chimes in?

Sounds like a plan!

Planning on merging this tomorrow if there are no objections!

One alternative might have been to recreate the enum with some kind of coercion to turn that enum in to a u16 value. This would keep the original readability while chasing out this dependency.
Avoiding magic numbers, even though HTTP codes are pretty darn well known, is nice.


Is 'aggregation_strategy' parameter available for token classification pipeline?

Question
Hi. I have question.
From HuggingFace Transformers documentation, they have 'aggregation_strategy' parameter in token classification pipeline. Link
Need to know in this library provide this parameter?
Thanks.

Hi there 👋 This isn't yet supported, but if a community member is interested in contributing, I'd be happy to add it!

Hi I had implemented some helper functions based on the original tranformers aggregation functions to do simple and max aggregation strategies:

You can adapt it to do other kinds of aggregation, but in my point of view the max is the most usefull one.

Code implementation
type AggregationGroup = {
    score: number;
    entityGroup: string;
    tokens: TokenClassificationSingle[];
    word: string;
    start: number;
    end: number;
};

function maxAggregation(tokens: TokenClassificationOutput) {
    const grouped: AggregationGroup[] = tokens
        .filter((token) => !token.entity.startsWith("O"))
        .reduce((groups, current) => {
            const isBeginToken = current.entity.startsWith("B-");
            const isWordFragment = current.word.startsWith("##");

            if (isBeginToken && !isWordFragment) {
                return [
                    ...groups,
                    {
                        score: current.score,
                        entityGroup: current.entity.replace("B-", ""),
                        word: current.word,
                        tokens: [current],
                        // It should be current.start but 'start' is useless in transformers.js
                        start: current.index,
                        end: current.index,
                    } satisfies AggregationGroup,
                ];
            }

            const lastEntry = groups.pop();

            if (!lastEntry) {
                return groups;
            }

            // Discard if index distance is too far: n+1
            if (lastEntry.end + 1 < current.index) {
                const result = [...groups, lastEntry];

                if (isBeginToken) {
                    result.push({
                        score: current.score,
                        entityGroup: current.entity.replace("B-", ""),
                        word: current.word.replace("##", ""),
                        tokens: [current],
                        // It should be current.start but 'start' is useless in transformers.js
                        start: current.index,
                        end: current.index,
                    } satisfies AggregationGroup);
                }

                return result;
            }

            const tokens = [...lastEntry.tokens, current].sort(
                (a, b) => a.score - b.score
            );

            const maxToken = tokens.at(-1);

            const word = lastEntry.word.concat(
                // Include '##' means that word is part of previous, otherwise we need to add a blank space between
                isWordFragment ? current.word.replace("##", "") : " " + current.word
            );

            return [
                ...groups,
                {
                    ...lastEntry,
                    entityGroup:
                        maxToken?.entity.split("-").pop() || lastEntry.entityGroup,
                    score: maxToken?.score || lastEntry.score,
                    word,
                    tokens,
                    end: current.index,
                } satisfies AggregationGroup,
            ];
        }, new Array<AggregationGroup>());

    console.log(grouped);
    console.log(
        "MAX:",
        grouped
            // Only for my specific model in Portuguese, here I'm just filtering where is PERSONs
            .filter((group) => group.entityGroup === "PESSOA")
            .map((group) => group.word)
    );

    return grouped;
}
function simpleAggregation(tokens: TokenClassificationOutput) {
    const grouped = tokens
        .filter((token) => !token.entity.startsWith("O"))
        .reduce((groups, current) => {
            if (current.entity.startsWith("B")) {
                return [
                    ...groups,
                    {
                        score: current.score,
                        entityGroup: current.entity.replace("B-", ""),
                        word: current.word,
                        tokens: [current],
                        start: current.index,
                        end: current.index,
                    } satisfies AggregationGroup,
                ];
            }

            const lastEntry = groups.pop();

            if (!lastEntry) {
                return groups;
            }

            // Discard if is not same Entity Group of last entry
            if (lastEntry.entityGroup !== current.entity.replace("I-", "")) {
                return [...groups, lastEntry];
            }

            const tokens = [...lastEntry.tokens, current];

            const score = tokens.reduce(
                (max, token) => Math.max(max, token.score),
                -Infinity
            );

            const word = lastEntry.word.concat(
                // Include '##' means that word is part of previous, otherwise we need to add a blank space between
                current.word.includes("##")
                    ? current.word.replace("##", "")
                    : " " + current.word
            );

            return [
                ...groups,
                {
                    ...lastEntry,
                    score,
                    word,
                    tokens,
                    end: current.index,
                } satisfies AggregationGroup,
            ];
        }, new Array<AggregationGroup>());

    console.log(grouped);
    console.log(
        "SIMPLE:",
        grouped
            .filter((group) => group.entityGroup === "PESSOA")
            .map((group) => group.word)
    );

    return grouped;
}


[DS-4947] Add negative user sampling to evaluation

Describe the big picture - why are we doing this?
This is the (hopefully) final PR in adding the ability to negative sample users in collie models! Specifically, this is the PR that adds the ability to metrics.py. I also caught a couple other places in utils.py and cross_validation.py that needed updating as well. Please take your time reviewing this PR, it was the most difficult of the 3 for me.
Any additional details to clarify code in the PR?
The metrics code can be hard to follow at times, so as always, I am open to feedback in enhancing the readability with any refactoring.
How is this tested?
On an EC2 instance all the tests passed. We have one final ticket for this work where I will rerun all the tutorial notebooks with negative user sampling.
Pull Request Checklist

[x] Pull request includes a description of why we are doing this
[x] CHANGELOG has been updated
[x] Version in _version.py has been updated
[x] All tests in the tests folder pass with a local build
[ ] README has been updated (if applicable)
[ ] Documentation in docs has been updated (if applicable)
[ ] requirements.txt and requirements-dev.txt have been recompiled (if applicable)
[x] Docker image can be built using docker build -t collie .

Screenshots or GIFs?

Keeping an eye on the tests, but this is otherwise ready for re-review.

WOO tests fixed! Thanks for the help @nathancooperjones , back to you!


Add informative Errors

Right now we're just using strings

Hi, I'd love to help with this if I can, do you have anything in mind for the errrors. I've been reading through the repo and it looks really cool

Do you have experience using thiserror?
I haven't put a ton of thought into it, but essentially I was just gonna replace the error type in crate::error::Result with some empty enum and let the compiler tell me what to fix. This is a "good first issue" but it might be a bit of busy work.
The one thing I want to stress is that the domain should define the shape of the errors, but allow the implementers of the ports to extend the error types. Something like:
#[derive(Error, Debug)]
pub enum Error {
    #[error("Not enough {0} to fulfill transaction]
    InsufficientFunds(Policy),
    ...
    TxoRecord(TxoRecordError),
    ...
}

pub enum TxoRecordError {
    #[error("Failed to retrieve UTxOs: {0}")]
    FailedToRetrieveUTxOs(Box<dyn Error>),
    ...
}

...

So, essentially, the interfaces, like TxORecord or ValidatorCode will each have their own error type with some flexibility for the implementers, and then those can each be included in a set of core, domain errors.
Idk if that makes sense. I'm happy to give more examples. Just writing this on the fly.

Also open to ways we could chunk up the work into multiple PRs.

So, essentially, the interfaces, like TxORecord or ValidatorCode will each have their own error type with some flexibility for the implementers, and then those can each be included in a set of core, domain errors.
Idk if that makes sense. I'm happy to give more examples. Just writing this on the fly.

I think I get what you mean, have on main Error enum and it should have variants like TxORecord(TxORecordError) where the TxORecordError type allow for more flexibility when adding more error cases for those specific areas? Also I wanted to ask what exactly TxORecord is meant to represent, I guessed an onchain address but not sure.

Also I wanted to ask why Policy is an Option<Address>

TxORecord is my name for the abstraction of the Blockchain. I don't think it's a great name, but I just chose something (I'm open to suggestions). I didn't go with Blockchain because that's just an implementation detail; the domain of Naumachia doesn't care about it being a blockchain or a centralized ledger. All it cares about is that there are UTxOs owned by different addresses, essentially. Hence TxORecord.
So, currently, I just have an InMemoryRecord and a LocalPersistedRecord that are useful for tests and for writing front-ends without reaching out to the chain.

I think I get what you mean, have on main Error enum and it should have variants like TxORecord(TxORecordError) where the TxORecordError type allow for more flexibility when adding more error cases for those specific areas?

Sounds like we're on the same page. Each interface can have its own error and that way the implementer of that interface can add relevant errors that the core domain can't know about. i.e. The LocalPersistedRecord would return file-system errors, etc.

Also I wanted to ask why Policy is an Option

I also considered creating my own enum for Policy:
enum Policy {
ADA,
NativeToken(Address)
}
But it's kinda weird, since ADA is just "", so maybe neither is right.
This doesn't even cover TokenName stuff, which Naumachia has no concept of yet.
I want to land on whatever is easiest to reason about within someone's SC. "Wishful driven design." As long as the information is conserved between conversions, of course--at the end of the day, it needs to be able to get converted to the on-chain representation.
As you can tell, a lot of this is in flux, but I've gone out of my way to put things behind interfaces and aliases so changing them in the future will have minimal impact.

I just did a search of all the traits.
Probably SCLogic needs its own error.
SmartContractTrait is a little different since that's the trait we provide to our consumers to mock Naumachia as a whole. It should should use the core error for that reason.

I'm thinking of just modelling the interface errors after the trait functions following your example.

Awesome. And if we decide we want something different in the future we can change it.
The library is still very much in exploration mode. I just wanted to get a working prototype out to the world so people could see what I had in mind and start contributing.

I'd recommend having the trait errors live with the traits, btw.

I wanted to ask if this looks right
#[derive(Debug, Error)]
pub enum TxORecordError {
    #[error("Insufficient amount of {0:?}")]
    InsufficientAmountOf(Policy),
    #[error("Failed to retrieve outputs at {0:?}: {1:?}")]
    FailedToRetrieveOutputsAt(Address, Box<dyn std::error::Error>),
}

I think so. I'd have to see there specific uses. Feel free to put a PR up and I can see how you're using them.
You could prolly get away with doing just one of the trait errors for now, just as a test run. You'll just need to .map_err them into the core error type.

Also not sure how to type the fields of the SCLogicError as it's generic
#[derive(Debug, Error)]
pub enum SCLogicError {
    // TODO Don't know how to constrain the type for these fields
    #[error("Failed to handle endpoint {0:?}")]
    FailedToHandleEndpoint(Box<dyn error::Error>),
    // TODO Not sure if this is right but not fully sure how the error for a failed lookup should look
    #[error("Failed to lookup: {0:?}")]
    FailedToLookup(Box<dyn error::Error>)
}

I will send a pr but here's an example of me using the TxORecordError in the select_inputs_for_one method
        for (policy, amt) in values.iter() {
            if let Some(available) = address_values.remove(policy) {
                if amt <= &available {
                    let remaining = available - amt;
                    remainders.push((remaining, address.clone(), policy.clone()));
                } else {
                    return Err(Error::TxORecord(TxORecordError::InsufficientAmountOf(
                        policy.to_owned(),
                    )));
                }
            } else {
                return Err(Error::TxORecord(TxORecordError::InsufficientAmountOf(
                    policy.to_owned(),
                )));
            }
        }
        let other_remainders: Vec<_> = address_values
            .into_iter()
            .map(|(policy, amt)| (amt, address.clone(), policy))
            .collect();
        remainders.extend(other_remainders);
        Ok((all_available_outputs, remainders))
    }

Also not sure how to type the fields of the SCLogicError as it's generic

Ah. Prolly use an assoc type for that as well. I'm getting a bit of a smell from all the types on the trait... but still seems right for now.
So you don't need to write that Error... but each of the impls will need one added.

I will send a pr but here's an example of me using the TxORecordError in the select_inputs_for_one method
Sweet. Yeah. this is essentially what I had in mind for now.

Also had slight issue backend module, will just create a Box<dyn std::error::Error> for the error not quite sure what is going in this snippet:

Not sure if this is ideal

Also had slight issue backend module, will just create a Box for the error not quite sure what is going in this snippet:

This is just saying that there is no From impl for those traits. The return Error type doesn't match the expected Error type for that function.
At least, I think that's what's happening on quick glance.

Not sure if this is ideal

It's looking pretty good. Is there something I'm missing :P

Also had slight issue backend module, will just create a Box for the error not quite sure what is going in this snippet:

This is just saying that there is no From impl for those traits. The return Error type doesn't match the expected Error type for that function.
At least, I think that's what's happening on quick glance.

Sorry didn't give context. when I use the error type I defined like so:

It throws an error

Yeah. You're trying to pass the Output as an dyn Error. Instead, say:
    InputDoesNotExist(Output<Datum>)

Or, to make your life easier:
   InputDoesNotExist(String)

and just include the tx_i.id().


invsim_apikey's error

I'm facing this issue sometimes, but my invsim_apikey's is filled correclty and all my selected skins are showing up for me
Error log: [EROR] (plugin:InventorySimulator) POST /api/increment-item-stattrak failed, check your invsim_apikey's value.

Well, there isn't much I can do besides adding the full URL in the debug message. Check the URL it's hitting when you get that error again. The code specifically looks for 401 error code to print that message, so your API is returning that if you confirm it's your URL.


Pass through all errors in both Stream::take and Stream::skip until stream is done.

Fixes issue #188.

Thanks @tmiasko!