HttpRequest.replaceHeader(myHeader) is not replacing an existing header, is creating new expectation

I've the following method
public MockHelper setupForwardAll() {
        mockServerClient
                .when(request())
                .forward(
                        forward()
                                .withHost(hostToForward)
                                .withPort(443)
                                .withScheme(HTTPS)
                );
        return this;
    }

that have the aim to forward all the request if doesn't match any expectations I've previously set, so I don't need to mock everything in my tests (not the best approach maybe, but it's not the point).
If I do this tiny modification:
public MockHelper setupForwardAll() {
        Header hostHeader = new Header(HOST.toString(), hostToForward);
        mockServerClient
                .when(request().replaceHeader(hostHeader))
                .forward(
                        forward()
                                .withHost(hostToForward)
                                .withPort(443)
                                .withScheme(HTTPS)
                );
        return this;
    }

I'm expecting that mockserver is going to replace the header "Host" (which value is my local ip machine because I'm redirecting all network requests to the mockserver) to the real host, but what actually it's doing is creating an extra expectation. See the following log:
2018-05-03 13:39:18,187 INFO o.m.m.HttpStateHandler request:
2018-05-03T13:39:18.201293000Z
{
  "method" : "GET",
  "path" : "/foo/bar",
  "headers" : {
    "Authorization" : [ "Bearer someToken1234" ],
    "content-length" : [ "0" ],
    "Accept" : [ "application/json" ],
    "User-Agent" : [ "node-superagent/3.5.2" ],
    "Base-User-Id" : [ "0" ],
    "host" : [ "<IP_ADDRESS>:1080" ],
    "connection" : [ "keep-alive" ],
    "accept-encoding" : [ "gzip,deflate" ]
  },
  "keepAlive" : true,
  "secure" : false
}
2018-05-03T13:39:18.201497500Z
 did not match expectation:
2018-05-03T13:39:18.201517000Z
{
  "headers" : {
    "Host" : [ hostToForward ]
  }
}
2018-05-03T13:39:18.201584700Z
 because:
2018-05-03T13:39:18.201603800Z
method matches = true
path matches = true
query string parameters match = true
body matches = true
headers match = false
cookies match = true
keep-alive matches = true
ssl matches = true

Extra information:
Mockserver version: 5.3.0

replaceHeader(hostHeader) replaces the header in the HttpRequest object created with the request() factory method.  The forward action you have specified doesn't update the request being forwarded in any way.  If you want the request being forwarded to be update try using one of the forward actions that updates the request, there are 5 options to choose between as detailed on the website: http://mock-server.com/mock_server/creating_expectations.html#actions

Thanks for the insight, it absolutely worked 😄


Initial draw is different?

I don't know if this belongs here but too me the inital draw on all controls, especially seen on DataGridView is much slower (multiple paints / flickering) on the .NET5 variants.
ManyControls for example also flickers more.
Do you experience this too?

The code here is contrived (refreshing many, many times) to help get a more accurate view the allocation differences and it isn't using double buffering so flickering is expected. Rendering is actually faster which makes the flickering more noticeable.

The code here is contrived (refreshing many, many times) to help get a more accurate view the allocation differences and it isn't using double buffering so flickering is expected. Rendering is actually faster which makes the flickering more noticeable.


Handle attributed C-Sets with nauty

In principle it should be possible to encode an attributed C-Set as a graph for nauty to consume. We would just put all distinct attribute values into singleton partitions that are already ordered in the nauty input by the inherent < ordering of the attribute.

addressed by https://github.com/AlgebraicJulia/CSetAutomorphisms.jl/pull/28


⚠ Bofrost.be has degraded performance

In cb44feb, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 955 ms

Resolved: Bofrost.be performance has improved in 408d214 after 6 minutes.


Capture all host logs for auditing purposes

In order have all platform logs available for auditing as Cloud Operations I want to send bosh job logs to CloudWatch.
Acceptance Criteria

[ ] /var/vcap/* is sent to CloudTrail via AWS Logs

Implementation Sketch

Add a pre-start job configuration to our awslogs release that

iterates over something like /var/vcap/sys/logs/*/*
and generates an agent config file to send all discovered logs to cloudwatch

Seems like this story needs an implementation sketch before we could think about moving forward on it. So... how do we propose to actually do this? I see a few possibilities:
Maintain an exhaustive list of logs to capture in the release
This seems to be what we are doing today. We'd just need to list everything else here and commit to keeping it up-to-date.
Modify our other releases to declare which logs they generate should be archived
See Compartmentalizing CloudWatch Logs Agent Configuration Files for an idea of how we could do this.  We could have each release / job write a configuration file out declaring which of it's logs should be backed up.
Modify the awslog startup scripts to generate a configuration on boot
I'm envisioning we'd have a small script that would iterate over something like /var/vcap/sys/logs//.log and append each file to the awslogs configuration before start.
Thoughts? Better Ideas?

Testing more comprehensive log capture on a test host.

Committing my initial attempt at the pre-start script: https://github.com/18F/cg-awslogs-boshrelease/pull/5
Needs more work tomorrow, tho --

The above PR seems ready to me. If @cnelson approves and merges, I can test and then push this to the next column.

Changes merged to master -- now verifying we are seeing new log events in AWS console.

Deployed awslogs changes to logsearch --

Didn't work because the additional config dir was being improperly specified.

Okay, in testing it looks as if {instance_id} is not unique enough and is causing zillions of this error:
cwlogs.push.publisher - WARNING - 12345 - Thread-67 - Multiple agents might be sending log events to log stream (nats_to_syslog i-a89101112) with sequence token (LONG_STRING_OF_DIGITS|none). This could cause duplicates and is not recommended.

I might need to amend the log stream identifier, or there might be another problem.

Okay, after testing different log_stream_name configs (didn't have much hope for that), I conclude that the aggressive harvesting of all files under /var/vcap/sys/log is causing sequence token errors. I'm going to be smarter about which files to ingest in the pre-start script, because gathering all valid logs for compliance is the most important goal here.

Okay, I have a unique log stream name & a sane group name and am excluding very obviously non-ingestable logs from the config (files that are not text or empty, and files that have a rotation timestamp) to avoid uselessly spawning log readers. No sequence errors nor multiple-agent errors any more.

Just waiting on a thumbs-up and merge so we can redeploy and test.

Waiting on merge -- https://github.com/18F/cg-awslogs-boshrelease/pull/7

I'm seeing logs with the new log stream name pattern everywhere I look in CloudWatch. I'm just not sure if there's any deploy I need to manually kick off to be sure we're collecting everything everywhere, but this looks good to me.

@sharms will accept later today

The current implementation triggers on pre-start which means that it will be executed before any jobs start.  The current code will not add any logs during this time, as it searches for log files which have been written to (1), which do not exist on a stemcell upgrade.
This worked on staging as these systems are frequently rebooted without cause, however it is rare we reboot production.
Recommendations:

Move log detection into startup script

https://github.com/18F/cg-awslogs-boshrelease/blob/master/jobs/awslogs/templates/bin/pre-start#L9

Good catch!  My bad for suggesting pre-start.

@datn Did you address Steve's feedback about this running in pre-start?
I'm not sure what the best approach is here, but I can see  a few options:

Toss a fairly long sleep() into the start of this job to give everything else a chance to start before we scan for logs
Move this code to a post-start job and extend it to also restart/HUPs the agent?
Inject a cron job to scan for new logs and restart the agent any time it finds them so we continually add new logs as they are created

Post-deploy might be a better place to move this, since we'd want all jobs to have started before we set up awslogs: https://bosh.io/docs/post-deploy.html. Or a cron job.

I missed the notification that this went back into ready, will look at it and everyone's helpful suggestions today.

I'm not understanding the problem @sharms points out, so I'll ask for clarification in channel.

https://github.com/18F/cg-awslogs-boshrelease/pull/11

Tested on admin-ui in staging, and while new logs are being added to the awslogs config file, they do not appear to be making it to cloudwatch.
I think the problem is that monit is not in the root user's path by default, so this call to restart isn't executing.

https://github.com/18F/cg-awslogs-boshrelease/pull/12 uses a new approach to ensure logs are captured even on short-lived / ephemeral VMs

@datn can you accept this one?

Accepting this one.


Fix SASS linter warnings

This commit fixes most SASS linter warnings, more precisely 154
warnings.  There are 2 types of issues not yet solved, which are a bit
more complicated:

IdSelector: Avoid using id selectors
NestingDepth: Nesting should be no greater than 5, but was 6

These might have impact in the layout, so they need a bit more testing.
I will try to fix those in a separate commit later.
The aim is to then add govuk-lint-sass to the Jenkinsfile and make
sure we keep out SASS files consistent with the guidelines.

LGTM :+1:


presidio-ocr fails building if tesseract libs are missing

Describe the bug
Building binaries (make build) fails if tesseract libs are not available.
To Reproduce
Steps to reproduce the behavior:

Build using makefile

$ make build
go build -ldflags ' -X github.com/Microsoft/presidio/pkg/version.Version=v0.3-21-g78be69b' -o bin/presidio-anonymizer ./presidio-anonymizer/cmd/presidio-anonymizer
go build -ldflags ' -X github.com/Microsoft/presidio/pkg/version.Version=v0.3-21-g78be69b' -o bin/presidio-ocr ./presidio-ocr/cmd/presidio-ocr
# github.com/Microsoft/presidio/vendor/github.com/otiai10/gosseract
tessbridge.cpp:5:10: fatal error: tesseract/baseapi.h: No such file or directory
    5 | #include <tesseract/baseapi.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:26: presidio-ocr] Error 2

See error

Expected behavior

State the dependency in the Development docs.
Explicitly add this dependency, if possible.

Not sure if (2) is doable, though. For Fedora 30 I fixed this by installing the tesseract-devel package. For Ubuntu the libtesseract-dev package can be used. (Source: https://github.com/otiai10/gosseract/issues/132#issuecomment-519427089).

Sorry, missed a line in the Development notes :-( (nr 4 at https://microsoft.github.io/presidio/development.html)


display-manager: fix argument handling of sddm

Motivation for this change
previously session type was not correctly set.
Currently only tested with session type 'none + awesome wm' and ssdm.
Other combinations should be handled in the same way, but for
critical components like login manager, be better safe then sorry

so please test this with your setup.

TODO:

[ ] run integration tests

Things done

[x] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

I'm also using awesomewm, and this fixes SDDM for me. :+1: for port-to-stable (or at least 17.03).

Still a WIP?

@Mic92 Fixed i3 for me. Thank you!

I reverted this on the 17.03 branch because it breaks slim: 99dfb6dce37edcd1db7cb85c2db97089d9d5f442.

@edolstra That's unfortunate... Does it break slim on the master branch as well (IIRC slim doesn't get any updates anymore)? And do we know the exact xsession parameters when called from slim?
Regarding my setup: For some very strange reason SDDM stopped providing any parameters (only $0 (obviously)) when calling the xsession script. I have no idea how this could happen (I don't remember changing my configuration or nixpkgs).
In order to see the parameters I've inserted:
      >&2 echo "xsession-parameter:"
      >&2 echo "$0"
      while [ $# -ge 1 ]; do
        >&2 echo "$1"
        shift
      done
      >&2 echo "xsession-parameter(end)"

At the following file: nixos/modules/services/x11/display-managers/default.nix
Directly after the following lines:
  # file provided by services.xserver.displayManager.session.script
  xsession = wm: dm: pkgs.writeScript "xsession"
    ''
      #! ${pkgs.bash}/bin/bash

Output from journalctl --boot:
-- Logs begin at Wed 2017-03-29 23:25:46 CEST, end at Sat 2017-04-22 23:23:56 CEST. --
[...]
Apr 22 23:23:39 localhost sddm-greeter[958]: Reading from "/nix/store/9bffhwsxn58k0rkj7idny1b66yci69sx-desktops/none + i3.desktop"
Apr 22 23:23:39 localhost sddm[937]: Message received from greeter: Login
Apr 22 23:23:39 localhost sddm[937]: Reading from "/nix/store/9bffhwsxn58k0rkj7idny1b66yci69sx-desktops/none + i3.desktop"
Apr 22 23:23:39 localhost sddm[937]: Reading from "/nix/store/9bffhwsxn58k0rkj7idny1b66yci69sx-desktops/none + i3.desktop"
Apr 22 23:23:39 localhost sddm[937]: Session "/nix/store/9bffhwsxn58k0rkj7idny1b66yci69sx-desktops/none + i3.desktop" selected, command: "/nix/store/465cd4i3w615vbhznpl4pv3c2hmpa1ab-xsession 'none + i3'"
Apr 22 23:23:39 localhost sddm-helper[1000]: [PAM] Starting...
Apr 22 23:23:39 localhost sddm-helper[1000]: [PAM] Authenticating...
Apr 22 23:23:39 localhost sddm-helper[1000]: [PAM] Preparing to converse...
Apr 22 23:23:39 localhost sddm-helper[1000]: [PAM] Conversation with 1 messages
Apr 22 23:23:39 localhost sddm-helper[1000]: [PAM] returning.
Apr 22 23:23:39 localhost sddm[937]: Authenticated successfully
Apr 22 23:23:39 localhost sddm-greeter[958]: Message received from daemon: LoginSucceeded
Apr 22 23:23:39 localhost sddm-helper[1000]: pam_unix(sddm:session): session opened for user john by (uid=0)
Apr 22 23:23:39 localhost systemd[1]: Created slice User Slice of john.
Apr 22 23:23:39 localhost systemd[1]: Starting User Manager for UID 1001...
Apr 22 23:23:39 localhost systemd[1001]: pam_unix(systemd-user:session): session opened for user john by (uid=0)
Apr 22 23:23:39 localhost systemd-logind[920]: New session 2 of user john.
Apr 22 23:23:39 localhost systemd[1]: Started Session 2 of user john.
Apr 22 23:23:39 localhost sddm-helper[948]: [PAM] Closing session
Apr 22 23:23:39 localhost sddm-helper[948]: [PAM] Ended.
Apr 22 23:23:39 localhost sddm[937]: Auth: sddm-helper exited successfully
Apr 22 23:23:39 localhost sddm[937]: Greeter stopped.
Apr 22 23:23:39 localhost systemd[1001]: Reached target Timers.
Apr 22 23:23:39 localhost systemd[1001]: Reached target Sockets.
Apr 22 23:23:39 localhost systemd[1001]: Reached target Paths.
Apr 22 23:23:39 localhost xsession[1009]: xsession-parameter:
Apr 22 23:23:39 localhost xsession[1009]: /nix/store/465cd4i3w615vbhznpl4pv3c2hmpa1ab-xsession
Apr 22 23:23:39 localhost xsession[1009]: xsession-parameter(end)
Apr 22 23:23:39 localhost systemd[1001]: Reached target Basic System.
Apr 22 23:23:39 localhost systemd[1001]: Starting SSH Agent...
Apr 22 23:23:39 localhost systemd[1001]: Started SSH Agent.
Apr 22 23:23:39 localhost systemd[1001]: Reached target Default.
Apr 22 23:23:39 localhost systemd[1001]: Startup finished in 10ms.
Apr 22 23:23:39 localhost systemd[1]: Started User Manager for UID 1001.
Apr 22 23:23:39 localhost sddm-helper[1000]: Starting: "/nix/store/465cd4i3w615vbhznpl4pv3c2hmpa1ab-xsession /nix/store/465cd4i3w615vbhznpl4pv3c2hmpa1ab-xsession 'none + i3'"
Apr 22 23:23:39 localhost sddm-helper[1009]: Adding cookie to "/home/john/.Xauthority"
Apr 22 23:23:39 localhost sddm[937]: Session started
Apr 22 23:23:39 localhost xsession[1009]: xsession-parameter:
Apr 22 23:23:39 localhost xsession[1009]: /nix/store/465cd4i3w615vbhznpl4pv3c2hmpa1ab-xsession
Apr 22 23:23:39 localhost xsession[1009]: xsession-parameter(end)
[...]

But it looks like the script should've been called with parameters: command: "/nix/store/465cd4i3w615vbhznpl4pv3c2hmpa1ab-xsession 'none + i3'"

I just noticed that we're actually using the wrong quoting within our desktop entry files.
#25138 should fix that (but in order to test this I need to fix my other bug first). This might fix this SDDM issue as well.


Support for pre fetch

Support for https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/prefetch-src

Thanks


KeyError 'refdoc'

I am getting a KeyError on the key 'refdoc' in ablog.  See the attached error file.
sphinx-err.txt
Is there a work-around?

I cannot reproduce this. Can you copy the command that you run? Is your project available on GitHub for me to reproduce it?

I don't have a convenient way to get you the source.  It is a main directory and conf.py linked to all of my projects.  I use Sphinx + ABlog to make my website (http://timcera.bitbucket.org/).  I use several Sphinx extensions and by commenting out each in turn found that the conflict is with 'sphinx.ext.autodoc' and 'sphinx.ext.autosummary'.
The command that I run is 'make html' which equates to 'sphinx-build -b html -d build/doctrees   source build/html'.

A new version of Ablog (v0.9.0) is out at this repo.
This issue should be fixed now but we made it explicitly support Sphinx>=1.6.


JBPM-5438: Stunner - get DataTypes for Data I/O Editor & Process Vars from server

@jlindop I think StunnerFindDataTypesQuery should be renamed to FindDataTypesQuery and reside in kie-wb-common-refactoring-backend/ with the other queries (you should rename the one in jbpm-designer to DesignerXYZQuery (and update that where its currently used) so they don't have the same name.
Also DataTypesQueryService would be better to be called FindDataTypesService (or similar) as it could be extended to run more than just a single query in future.
just my 2c
otherwise +1

@tsurdilo Thanks for the suggestions - I've made those changes. The Designer query is already called DesignerFindDataTypesQuery, so there won't be a conflict.

@jlindop thanks
+1


[R4R]: remove redundant check

Goals of PR
Core changes:

remove redundant check

Notes:

Write notes here

Related Issues:

Link issues here

Maintain the current state and make no modifications


Support for mypy's new semantic analyzer

I'm trying to upgrade to mypy==0.740 which now apparently has a new semantic analyzer. It seems to trigger a lot of errors that weren't there using 0.710. The mypy docs mention that some plugin-related things have changed (mainly the order in which AST nodes are processed).
I was wondering if anyone else has been using sqlalchemy-stubs with mypy 0.720+ (which made the new semantic analyzer default), and if they were succesful.
I couldn't figure out from the travis build which mypy versions are supported for sqlalchemy-stubs.

What kind of errors do you see and why do you think they are related to this plugin?
(We are using a relatively recent version of sqlalchemy-stubs with a very recent version of mypy.)

One where I think sqlalchemy-stubs might be the culprit looks like this:
mypackage/schematic/element.py:61: error: Name '__init__' already defined on line 52
        def __init__(self, *args, **kwargs):
        ^

Where line 52 is:
class SchematicElement(Model):

Where Model is defined as (in another module):
from sqlalchemy.ext.declarative import declarative_base

Model = declarative_base()

Another error we get a lot now is for every relationship call we have:
mypackage/dashboard.py:118: error: Cannot find model "mypackage.user.User"
        user = relationship("mypackage.user.User")
               ^

In previous versions of mypy this would work, as we would import all models during typecheck using this at the top of the file:
if TYPE_CHECKING:
    from .db import *  # NOQA: F401, F403

Where db.py contains from module import * imports for all our modules containing our models (with valid __all__ definitions in all those modules.

The __init__() error is definitely a bug in the plugin, we give priority to user-defined __init__() for models if present, otherwise we insert an auto-generated one.
For the second one, the rules for star imports in mypy have been changed twice between 0.710 and now, in particular IIRC some names are considered private now, so the error may not be caused by the plugin. Just as a sanity check could you please add test_user: mypackage.user.User; reveal_type(test_user) just above the problematic line and see what mypy says?

Adding that gives me these lines:
mypackage/dashboard.py:119: error: Name 'mypackage' is not defined
mypackage/dashboard.py:120: note: Revealed type is 'Any'
mypackage/dashboard.py:121: error: Cannot find model "mypackage.user.User"

Note that I'm not getting the errors on 0.720, and they start appearing on 0.730.
(also on 0.730 setting new_semantic_analyzer=False in mypy.ini crashes mypy pretty spectacularly with an AttributeError, but that seems to be fixed on 0.740)

Adding that gives me these lines: [...]

Then as you can see this is not caused by the plugin, and you might need to adjust your __all__ definitions. Also you might want to try mypy master.

Ok, should I create an issue with mypy for that? mypy master throws the same errors. Did something change regarding implicit reexports?
We define our models in their own files:
mypackage/user.py
__all__ = ["User"]
class User(Model):
    ....

And then import * (implicitly re-exporting them again), in a single entrypoint:
mypackage/db.py
from .user import *

I think that re-export is where things fall apart, because when I just import from .user import User, the reveal_type call reveals mypackage.user.User (as expected).

Ok, should I create an issue with mypy for that?

You can try, but before that be sure you have a minimal repro (without plugins) that works in Python bu fails with --implicit-reexports, and you read this part of spec https://mypy.readthedocs.io/en/latest/command_line.html#cmdoption-mypy-no-implicit-reexport


[Enhancement] Add anchor not to be used on array/object

Description
Need to update the guidance based on newly-discovered issues that the add anchor should not be used on a list of objects like containers[] in a Pod.
Slack discussion
No response

foreach should be used in tandem with the add anchor but not the add anchor by itself.


Add traces with different levels

Currently there are 3 macros to trace information TRACE_LEVELXX (see trace.h). However they are not really used and there is a random mix between normal prints and this macros.
Task:

Use the different TRACE_LEVELXX macros according to the relevancy of what is being traces. Level 0 for important information, level 1 for more detailed stuff and level 2 for very detailed traces.
Replace the existing prints.

@alrevuelta
I need to rewrite the conv operator, since it produces wrong data in a network of mine.
Since this operator is large, I would rewrite the logging first to see what is happening inside.
My proposal:

all tracing/logging is done via macros, so we keep the binary clean if unneeded
file descriptor is settable for each severity

i.e. TRACE_INFO_FD

tracing can be overwritten per file

format: `TRACE_FILE="[:]{(;|)[:]}"
i.e. TRACE_FILE="inference.c:2;main.c:1"

tracing can be overwritten per function

format: `TRACE_FUNCTION="[:]{(;|)[:]}"
i.e. TRACE_FUNCTION="operator_xy:2 inference:1"

tracing is extended with node/attribute/tensor exports
exporting can be set globally/ per file/ per function

via TRACE_EXPORT, TRACE_EXPORT_FILE, TRACE_EXPORT_FUNCTION

single generic trace macro with fprintf style arguments

TRACE(level, file, format_string, ...)

various variants that use this generic trace macro

severity

TRACE_INFO(format_string, ...)

file, line, function
on TRACE_LEVEL >= 1
stdout

TRACE_DETAIL(format_string, ...)

file, line, function
on TRACE_LEVEL >= 2
stdout

TRACE_DEBUG(format_string, ...)

file, line, function
on TRACE_LEVEL >= 3
stdout

TRACE_WARN(format_string, ...)

file, line, function
on TRACE_LEVEL >= 1
stderr

TRACE_ERROR(format_string, ...)

file, line, function
on TRACE_LEVEL >= 1
stderr
flush

TRACE_FATAL(format_string, ...)

file, line, function
on TRACE_LEVEL >= 1
stderr
flush
exit

data types

TRACE_TENSOR(tensor)

name, shape, datatype on TRACE_LEVEL >= 1
values on TRACE_LEVEL >= 2
complete dump if TRACE_LEVEL >= 3

TRACE_ATTRIBUTE(attribute)

name, datatype on TRACE_LEVEL >= 1
values on TRACE_LEVEL >= 2
complete dump on TRACE_LEVEL >= 3

TRACE_NODE(node)

name, op_type, domain, inputs, outputs, attributes on TRACE_LEVEL >= 1
values on TRACE_LEVEL >= 2
complete dump on TRACE_LEVEL >= 3

misc

TRACE_ENTER(message)

file, line, function

TRACE_EXIT(message)

file, line, function

TRACE_CALL(func_pointer, ...)

file, line, function, args, return value

conditionals

all macros have a conditional variant

i.e. `TRACE_INFO_IF(condition, format_str, ...)

I need to rewrite the conv operator, since it produces wrong data in a network of mine.

Is it with mobilenetv2? I was playing with the onnx one and seems to fail because the group attribute is not implemented. There are many hardcodings so I wouldn't be surprised if its failing somewhere else. Respect if you manage to fully implement it.
I wrote something recently that can come in handy for debugging see this. Its a new feature that allows to dump all intermediate outputs to a file, and then assert the output each node individually to see which node is causing problems. Quick prototype though.
Regarding the tracing levels.

all tracing/logging is done via macros, so we keep the binary clean if unneeded
Totally agree.

Generally I like the idea but I would simplify some things:

I don't see the need of the traces under misc and conditionals
warn, error, and fatal ok.
info, detail, debug. Why all 3? Do we really need that granularity?
I liked the current idea of having TRACE_LEVEL0, TRACE_LEVEL1 and TRACE_LEVEL2.

So my idea would be something like:

Have TRACE_LEVEL0, TRACE_LEVEL1 and TRACE_LEVEL2 as the only macros that call printf.
Then have other macros that include the previous ones such as:

TRACE_WARN
TRACE_ERROR
TRACE_FATAL
TRACE_TENSOR
TRACE_ATTRIBUTE
TRACE_NODE

So when we want to trace something we can either call the direct TRACE_LEVELX function, or "call" another macro that offers some kind of extra functionality (i.e. TRACE_TENSOR that iterates the tensor and prints different information according to the trace level that is set).
I'm trying to simplify it because I don't want to end up with traces that we don't use, or that are used wrong. The less traces, the less we have to enforce that they are used correctly, because they would be easier to understand.

Is it with mobilenetv2? I was playing with the onnx one and seems to fail because the group attribute is not implemented. There are many hardcodings so I wouldn't be surprised if its failing somewhere else. Respect if you manage to fully implement it.

yeah with my mobilenetv2 network: conv produces wrong shapes (one dimension is off by one), but didn't look into it.
I'm already preparing for the nightmares regarding this operator :D

I'm trying to simplify it because I don't want to end up with traces that we don't use, or that are used wrong.

But by explicitly not stating what this trace_level is for, you are inviting misusage, don't you?
I just renamed your 0 to info, 1 to detail and 2 to debug and incremented them by one since TRACE == 0 should be no trace, shouldn't it?
the misc thing was for entering functions and leaving them, instead of having a funny trace call with a parameter equal to the __func__ macro value.
void operator_xy() {
  TRACE_ENTER(); //start of scope
  ...
  TRACE_EXIT(); //end of scope
  return 0;
}

I wrote something recently that can come in handy for debugging see this.

this is nice, will look into it.
exported pb files can be loaded with python via the onnx library.
I also wrote a utility function to generate these exports so I could compare them with the results of other frameworks.

I just renamed your 0 to info, 1 to detail and 2 to debug and incremented them by one since TRACE == 0 should be no trace, shouldn't it?

Ah okay, didn't saw that. Don't you prefer having 0, 1, 2 or 1, 2, 3? At least to me it looks more straight forward and I don't have to think, is detail more verbose than debug? Or it was the other way?
Regarding TRACE_ENTER and TRACE_EXIT. Why not just using debug or detail? Nothing really against, just trying to save some macros.

the conditionals are based on the idea that optional objects may have dependencies.
these macros allow to specify dependencies without writing weird if constructs which only apply to trace calls.

I'm afraid I don't understand the conditional macros. Could you elaborate?

Ah okay, didn't saw that. Don't you prefer having 0, 1, 2 or 1, 2, 3? At least to me it looks more straight forward and I don't have to think, is detail more verbose than debug? Or it was the other way?

fair enough :D
I would prefer names, at least in addition to numbers.
but you are right, debug may not have been the best choice.

Regarding TRACE_ENTER and TRACE_EXIT. Why not just using debug or detail? Nothing really against, just trying to save some macros.

The reasoning behind these was to minimize boilerplate
instead of
TRACE_INFO("enter function") and TRACE_INFO("exit function")
just
TRACE_ENTER() and TRACE_EXIT()
or  TRACE_START() and  TRACE_END()
this enforces a log format, making log searches easier.

I'm afraid I don't understand the conditional macros. Could you elaborate?

of course.
sometimes it's way more compact to have a conditional print.
for example, you have an optional attribute that overrides some default value, you can easily log this via
TRACE_INFO_IF(!attribute,"using default value %d\n",default_value);

instead of specifying an if/else case just for this.
this mechanism also allows for asserts:
TRACE_FATAL_IF(!attribute,"attribute not specified!\n");

and if we say that TRACE_{ATTRIBUTE,TENSOR,...} are fatal if they receive a NULL pointer
we can do the same thing even more compact:
TRACE_ATTRIBUTE(attribute)

would suffice.
if an attribute is optional we could always go back to a conditional variant:
TRACE_ATTRIBUTE_IF(attribute,attribute)

does not fail if attribute is not valid.
if we have an attribute bar that requires foo we can simply express this as
TRACE_ATTRIBUTE_IF(foo,bar)

The reasoning behind these was to minimize boilerplate
instead of
TRACE_INFO("enter function") and TRACE_INFO("exit function")
just
TRACE_ENTER() and TRACE_EXIT()
or TRACE_START() and TRACE_END()

one other option would be to use any supplied message as a log prefix.
for example:
TRACE_INFO("test");
// info:example.c:34::test
TRACE_START("block xy");
TRACE_INFO("test2");
// info:example.c:34:block xy:test
...

fair enough :D
I would prefer names, at least in addition to numbers.
but you are right, debug may not have been the best choice.

Nothing against but I still prefer numbers. It improves readability. But your choice :)
Lets keep discussing it once we have a PR in front.


Inside EditText

If i add a EditText into DialogPlus,then,
click editText -> pop soft keyboard -> click hardware BACK.
I want DailogPlus dismiss,but whole activity finished.
Is it a trouble?
Exp: your demo R.layout.content ,replace TextView with EditText

Yes, it's a bug which I recently faced as well. I'll try to find a solution for this. It happens when the dialog is displayed from fragment

Any solution for this issue? Its urgent.
Thanks in advance.

My way is that let Activity and Dialog coordinate.
One:  In Dialog holder class contain method check Dialog status.
eg: class SMSCodeGet {
static SMSCodeGet CODE_GET;// instance
DialogPlus dialogPlus;
public static boolean onBackPress() {
if (CODE_GET != null && CODE_GET.isShowing()) {// if dialog is showing ,close
CODE_GET.close();
return true;
}
return false;
}
}
Two: In Actvity'onBackPress call method onBackPress of SMSCodeGet.
eg:  @Override public void onBackPressed() {
if (SMSCodeGet.onBackPress()) {// if SMSCOdeGet consume backEvent, return
return;
}
super.onBackPressed();
}

Just add these codes to your activity
   @Override
    public void onBackPressed() {
        if (dialog.isShowing()) {
            dialog.dismiss();
        } else {
            super.onBackPressed();
        }
    }

There are some workarounds that is shown. For now I'll close this issue and re-open if this is still an issue in order to track it better.


Verification of Document Store Issuer in Goerli testnet invalid

Hi, I'm trying to test using a document store contract in the Goerli testnet to issue a wrapped document. However, I'm unable to verify the ISSUER_IDENTITY fragment. The document itself shouldn't have an issue because I can verify it in the Toolkit. Can anyone kindly point me in the right direction to do so?
Below is the document I'm trying to verify.
{
  "version": "https://schema.openattestation.com/2.0/schema.json",
  "data": {
    "$template": {
      "name": "b374f2a6-e35e-4c38-ae49-2d066646b201:string:main",
      "type": "640daddc-45d8-41b0-b069-95a0e3ac6f93:string:EMBEDDED_RENDERER",
      "url": "1d550033-5393-4350-ad75-22b319b229a2:string:https://tutorial-renderer.openattestation.com"
    },
    "recipient": {
      "name": "48c5eb06-2177-4dbc-bf44-16b652fd0ea1:string:John Doe"
    },
    "uselessField": "e22f6dd3-3a17-4e8b-930a-b7fd715b7d00:string:uselessString",
    "issuers": [
      {
        "name": "ea3583a0-efbb-4a90-a9c8-d2f597680355:string:Demo Issuer",
        "documentStore": "bd0f76e3-23b9-4461-9fc3-283d124fb607:string:0x1de923F8DB41a2FB35c03deB7Be8f85b134F1abE",
        "identityProof": {
          "type": "8c929c4a-92a2-45e2-be4e-5a94f1aa8b26:string:DNS-TXT",
          "location": "bc31c225-266a-4869-a330-d095ea16271e:string:remote-maroon-ant.sandbox.openattestation.com"
        }
      }
    ]
  },
  "signature": {
    "type": "SHA3MerkleProof",
    "targetHash": "c1b9eba14065631b463a4c0bf9a79460faa0074e76bdb848a2235c1a3d675c98",
    "proof": [],
    "merkleRoot": "c1b9eba14065631b463a4c0bf9a79460faa0074e76bdb848a2235c1a3d675c98"
  }
}

Below is the fragment that is invalid.
{
  "name": "OpenAttestationDnsTxtIdentityProof",
  "type": "ISSUER_IDENTITY",
  "data": [
    {
      "status": "INVALID",
      "location": "remote-maroon-ant.sandbox.openattestation.com",
      "value": "0x1de923F8DB41a2FB35c03deB7Be8f85b134F1abE",
      "reason": {
        "message": "Matching DNS record not found for 0x1de923F8DB41a2FB35c03deB7Be8f85b134F1abE",
        "code": 4,
        "codeString": "MATCHING_RECORD_NOT_FOUND"
      }
    }
  ],
  "reason": {
    "message": "Matching DNS record not found for 0x1de923F8DB41a2FB35c03deB7Be8f85b134F1abE",
    "code": 4,
    "codeString": "MATCHING_RECORD_NOT_FOUND"
  },
  "status": "INVALID"
}

As per the documentation, I'm providing the provider details in the environmental variables and using the default verify function. It works when I'm working with documents issued using DID but not with a Document Store.
PROVIDER_ENDPOINT_TYPE="jsonrpc"
PROVIDER_ENDPOINT_URL="https://goerli.infura.io/v3/<key>"
PROVIDER_API_KEY="<key>"

const { verify } = require("@govtechsg/oa-verify");

var fragments = await verify(document);

PROVIDER_ENDPOINT_TYPE="infura"
PROVIDER_ENDPOINT_URL="https://goerli.infura.io/v3/<key>"
PROVIDER_API_KEY="<key>"

I also tried to create a customized verify function using verificationBuilder and openAttestationVerifiers with the network name, but it still didn't work.
const { verificationBuilder, openAttestationVerifiers } = require("@govtechsg/oa-verify");

const verify = verificationBuilder(openAttestationVerifiers, { network: "goerli" });

Finally, I tried using the Provider option to create a customized verify function. Firstly with the Provider values in environmental variables, and later manually providing the Provider options but still no luck.
const { verificationBuilder, openAttestationVerifiers } = require("@govtechsg/oa-verify");

const provider = utils.generateProvider();
const verify = verificationBuilder(openAttestationVerifiers, { provider: provider });

const providerOptions = {
    network: "goerli",
    providerType: "infura",
    apiKey: "<key>",
  };
const provider = utils.generateProvider(providerOptions);
const verify = verificationBuilder(openAttestationVerifiers, { provider: provider });

able to share the repo to your application?

const provider = utils.generateProvider(providerOptions);

Sure. I have created another repo to simulate my issue. Below is the link.
https://github.com/jedtravis/openattestation-testing

your case 2 and case 4 is correct i think, u need to pass in the correct variable fragments2 and fragments4, not fragment.

I managed to find out what's the issue. My version was too old so goerli testnet support was not added yet. It worked after I changed it to the latest version. To be clear, other than case 1, all the remaining cases are correct. Thank you very much for your assistance!


Not working or not using correctly

I don't know how exactly this is supposed to work since there's no documentation, but when I click "Fill drive GUID" it says "Unable to get GUID." I've tried entering the drive letter first, I've tried finding the GUID manually and entering that in there, but nothing makes this plugin work.

Hello,
If you have a static drive (let's say a HDD bound to "D:") encrypted with bitlocker, you should  :

create a new entry, choose a title you want
put the password for the drive in the entry usual password field
Select "D:" as drive mountpoint in the KeeLocker tab

You can the either tick the checkbox "Unlock volume on opening" that will do so when the Database is opened (but that requires to close then open the DB). Or you can click "Unlock Volume" to test it right now.
If it is still not working : what flavor/version of windows are you using ?

Note  : the Drive GUID feature is for more complex scenarios.
Eg : you use the same DB on multiple machines, along with a USB stick cyphered using BitLocker. The issue is that on the first machine, you USB stick mounts as "D:", but on another one, it mounts as "E:". In such cases, the "Drive Mountpoint" is not super useful, so instead of using it, you can use the "GUID", that should work in such scenarios.

Yeah, I understand the purpose of GUID, and that's why I'm trying to use it, but I have no idea how. It doesn't autofill it (which is what it looks like it would do when clicking the button) and entering it manually doesn't work.

What flavor/version of windows are you using ?

Sorry. Win10x64 Pro 21H1 build 19043.1415

I've been testing on my end, with a very similar OS (Windows 10 Pro x64 21H1 build 19043.1706 ).
I created a USB stick :

With "Used Space" (recommended) encryption.
With the compatible mode OR with the new XTS-AES encryption. Both work.

When I choose the current drive mountpoint of a USB stick, and click "Fill drive GUID", it works. Unlocking also works.
The GUID should look something like : \\?\Volume{123456ab-1234-1234-1234-123456789abc}\
Do you see anything that could make a significant difference ?

Ok, that works, but the UI is very confusing and there are no instructions. The use of radio buttons suggests one or the other (which technically it is, but in this case you have to use one then the other, so while you're only using one in the end, you're still using both), so I didn't realize I had to select the drive in the top box then hit the button, the location of which directly under the bottom box makes it look like it's meant to be used with that box. I'd suggest a couple changes to make it more clear how to use it:

Put the "Fill drive GUID" button under the "Drive mountpoint" box (or better yet, next to it, since that box only needs to be about a tenth of its current length) and put some space between those two and the "Drive GUID" box
Add some brief instructions. For example, change "Use a GUID for removable drives that do not always mount on the same letter. Select a drive in "Drive mountpoint" dropdown and click "Fill drive GUID" button." You could also put this text in between the first dropdown and the button and second dropdown.

Unrelated, but, if possible, I'd also suggest having the drive dropdown show all drives, as it only shows A-H, and not all users may know they can simply type additional letters in.

I've tried to improve the UI in v1.3 based on your suggestions.
I have also renamed the button to "Convert to GUID", and it is now disabled unless in "mountpoint" mode. I believe it's now clearer and does not need explicit instructions.
I'd love your feedback though.
Thanks

It looks better, at least from what I can recall. I should have looked again at the old layout before updating, but this does seem to make at least more sense. That said, I already know how it works now, so I can't look at it with fresh eyes, and while maybe not necessary, there are still a few things that could be done to make it easier:

The main thing is I can still see users reading the note about the GUID and selecting that option directly, which then completely prevents them from being able to do it the proper way. They probably shouldn't be able to select it as a separate option, and it should be greyed out until they select a drive letter and click the button (I like the new label), at which point it gets filled in and un-greyed.
I still think it should list all drive letters, not just A-H, and perhaps allow just the letter or the letter and colon (e.g. D or D: instead of D:) and add the missing parts automatically. Not really necessary, but would just make it a bit more user-friendly and might help less technical people.
While not necessary if the first point is done, if it's not, I still think a bit more explanation would be useful, and certainly couldn't hurt. Having three lines of text instead of one-and-a-half isn't going to be crowded or overwhelming but very well could make the difference between somebody figuring out immediately how to use it vs having to play with it a bit or even look online to figure it out. Again, I can't look at it with a fresh perspective anymore, but I would probably figure it out much easier with the new layout; however, many users are not tech-savvy, and something as minor as another sentence of instructions to help those users is, IMO, well worth it. Again, though, not as necessary if the first point is addressed.

I think there are valid use-cases that are incompatible with the changes you are suggesting.

People can obtain GUID from other sources. There are cases when you don't even have to mount your drive, but still want to unlock it (with VMs/cross-compilation tools/WSL). I don't think it's a good idea to forbid people from entering a GUID to improve UX : this would make it harder to use in those cases.
The "Drive mountpoint" is a string (not just a drive letter). It's the very string I feed into the windows API. There are other syntaxes, like network shared storage, or drive mounted in a folder inside another drive. I don't want to temper with this string at all : there may be new syntaxes added in the future, or others I'm not aware of. Trying to be smart about this is bound to fail someday. The list of options is just here as a list of examples for this string, covering the most common cases.

That said :

I can add "..." at the end of the "Drive mountpoint" to make it more clear you can enter whatever you like
I can replace the text with
Use a GUID for removable drives that do not always mount on the same letter. You can obtain the GUID of your drive by selecting its current "Drive mountpoint" then clicking on "Convert to GUID"

Do you think this would make a significant improvement ?

I have been trying some improvements.

I will still have some ideas, but this is where I'm heading.

And the window look like that when the combobox is collapsed


Cortana with Outlook desktop is supported

Cortana with outlook is now supported and I have updated the document accordingly.
Problem: https://github.com/MicrosoftDocs/OfficeDocs-O365ITPro/issues/381

@nenonix @JohanFreelancer9 - kindly do the copy editor review for this PR. Thanks.

"You can add your Outlook or Office 365 account as a connected service to Cortana and add reminders and tasks to your lists in Microsoft To-Do. You can also use Cortana to read and search messages in the Outlook desktop app.

@joinimran - I will create a copy/edit for this that you can merge into this PR.

@kwekuako please review and decide whether to merge


cmd/compile: improve BCE to detect safe unbounded slicing

See https://github.com/golang/go/issues/31586#issuecomment-550481812
Core problem can be reduced to:
package scratch

import "encoding/binary"

func A1(b []byte) {
	key64 := uint64(1024)
	for len(b) >= 32 {
		v := binary.LittleEndian.Uint64(b)
		binary.LittleEndian.PutUint64(b, v^key64)
		v = binary.LittleEndian.Uint64(b[8:])
		binary.LittleEndian.PutUint64(b[8:], v^key64)
		v = binary.LittleEndian.Uint64(b[16:])
		binary.LittleEndian.PutUint64(b[16:], v^key64)
		v = binary.LittleEndian.Uint64(b[24:])
		binary.LittleEndian.PutUint64(b[24:], v^key64)
		b = b[32:]
	}
}

func A2(b []byte) {
	key64 := uint64(1024)
	for len(b) >= 32 {
		v := binary.LittleEndian.Uint64(b)
		binary.LittleEndian.PutUint64(b, v^key64)
		v = binary.LittleEndian.Uint64(b[8:16])
		binary.LittleEndian.PutUint64(b[8:16], v^key64)
		v = binary.LittleEndian.Uint64(b[16:24])
		binary.LittleEndian.PutUint64(b[16:24], v^key64)
		v = binary.LittleEndian.Uint64(b[24:32])
		binary.LittleEndian.PutUint64(b[24:32], v^key64)
		b = b[32:]
	}
}

BCE:
$ go build -gcflags="-d=ssa/check_bce/debug=1" scratch.go
# command-line-arguments
./scratch.go:10:34: Found IsInBounds
./scratch.go:12:34: Found IsInBounds
./scratch.go:14:34: Found IsInBounds

Bound checks remain in A1 even though they are not needed but are eliminated in A2 due to the explicit and not unbounded slices.
@renthraysk stated:

It was the _ = b[7] line in PutUint64() that was cause the bounds checks. The compiler failed to prove it's not needed with PutUint64(b[8:], x) however it does eliminate the first check in PutUint64(b, x).

I'm on 1.13.4 on macOS.

cc @zdjones

This is a duplicate of https://github.com/golang/go/issues/24876. Also see https://github.com/golang/go/issues/19126
The issue is that the prove pass does not currently do any arithimetic when assesing b[16:] and therefore has no information about the length of the resulting slice. see this comment: https://github.com/golang/go/issues/19126#issuecomment-372529793.
I have a chain of CLs that fix this, but they are not complete this cycle.

Here is the CL chain that fixes this issue: https://go-review.googlesource.com/c/go/+/190657

Closing as a duplicate.


Back button on mouse doesn't work

Issue Type: Bug
The mouse bouton "back" doesn't work on vscode
Actual result :
On vscode, when I press "Back" or "Next" button of my mouse, nothing happen. (keyboard shortcut "^control + -" works)
Expected result :
On vscode, when I press "Back" or "Next" button of my mouse, I should "Go back".
Hardware :
Laptop : MacBook Pro (14-inch, 2021), Chip Apple M1 Pro, MacOs 12.5
Mouse : Logitech Mx Anywhere 3 for Mac
FYI :
"Logi Option+ v1.11.1235" installed

VS Code version: Code 1.69.2 (Universal) (3b889b090b5ad5793f524b5d1d39fda662b96a2a, 2022-07-18T16:12:57.074Z)
OS version: Darwin arm64 21.6.0
Restricted Mode: No

System Info

Item
Value

CPUs
Apple M1 Pro (10 x 24)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmetal: disabled_offmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
2, 2, 2

Memory (System)
32.00GB (0.94GB free)

Process Argv
--crash-reporter-id cc740197-33a4-4372-bd97-5e3822b56902

Screen Reader
no

VM
0%

Extensions (6)

Extension
Author (truncated)
Version

Bookmarks
ale
13.3.1

githistory
don
0.6.19

gitlens
eam
12.1.2

git-graph
mhu
1.30.0

vsliveshare
ms-
1.0.5669

vetur
oct
0.35.0

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492:30256859
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonvspyt551cf:30345471
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30449410
vscscmwlcmt:30465135
cppdebug:30492333
pylanb8912:30529769
vsclangdc:30486549
c4g48928:30535728

Does information at #88029 help?

From the description the issue sounds like a setup / configuration problem outside of VSCode and nothing VS Code itself could fix.
VSCode is built on top of a large number of components that are not maintained by the VSCode team directly. This includes:

Electron as the cross platform UI framework
node.js for most integration into the OS
Chromium as runtime and rendering engine (HTML, CSS, JS)
and after all, the operating system VSCode is running on

We also have a large community over at https://stackoverflow.com/questions/tagged/visual-studio-code to ask for advice.
Happy Coding!


Add simple cloudy radiation example

This PR adds support for all-sky radiation to rrtmgp_model_callback. Specifically, it allows for cloud properties to be computed based on the thermodynamic state, or, if idealized_clouds == true, it puts an icy cloud between 4km and 5km, and it puts a wet cloud between 1km and 1.5km.

bors r+


Integrating with the front-end build process

The front-ends have standardized on a single setup.sh file that invokes npm, bower, and grunt. In regulations-site, we've integrated that with the Python build, so that a pip install will run the frontendbuild.sh script.
I'd like @Ooblioob and @rosskarchner's thoughts on whether this is particularly useful for delivery, as well as the rest of the team's thoughts on how we can make this reusable. The build_ext and bdist_egg overrides are required for build_frontend to be invoked by pip, and those overrides make it a bit harder to just wrap this in a separate package that can be imported.
It's not impossible — that package just provides those overrides too. But this could make the process more complex for any projects that need to also override parts of setuptools. Of course, those cases might be doing something so completely different it doesn't really matter.
Thoughts?

My first reaction, is that this is deeply weird-- but it's conceptually not that different from a Python module that includes C code, knowing how to compile itself.
This also relates to something @higs4281 is working through.
Just to make sure, even outside of the 'pip' context, a 'python setup.py install' will build the front end assets?
This might call for another discussion thread, but imagine we are distributing such an app on pypi: should the egg/wheel/whatever have this stuff already built?

One strategy could be to include all the built stuff in some kind of distribution directory and commit that to source control.  While I like the thought of only having to clone the code (and less setup), it pollutes our commit history in a way that makes reviewing PRs a nightmare.  Plus there would be lots of files that "we just have" where it's unsure why we have those files.
My worry with the frontendbuild.sh strategy is dependencies. For example, if frontendbuild.sh calls a grunt task, then grunt needs to be installed.  So does browserify, gulp and any other things we need to build, and it's not explicitly mentioned in our python setup.py file.  If our goal is that all applications should be able to build themselves, this might be less than ideal.
Granted, the alternative is we have ansible do the build, which also requires the dependencies of grunt, gulp, browserify, etc. But at least there we can ensure those dependencies are met.

@rosskarchner Yes, that's kind of what prompted this. @higs4281 and @marteki are pulling in what I did for regulations-site. I don't give any thought to using my own code, but when other people do it I tend to give it more thought ;)
The way this is set up right now, yes, python setup.py install will build the frontend assets. As to distributing... I'm not sure whether we should include those assets or not. Including generated files can be ugly (as @Ooblioob points out), and it seems a bit silly to ask the front-ends to commit their generated files, when if they came to us and asked if we could commit generated files so they could package things cleanly with npm, I'd be quite hesitant.
But yes, the front-end is generally assuming npm+brower+grunt. And this is including that in the process. I don't want to dictate possible dependencies to the front-end (IMHO they can build however they think it best to, it's their problem domain), but I don't want to introduce more back-end dependencies (like Ansible) because of this.
Perhaps the solution is something more to the effect of an npm, grunt, and bower build command that's not automatically executed. If we decided to package something for pypi we'd have to ensure that was done, but it wouldn't be automatic. That breaks the ability to just pip install one of our projects straight from GitHub, though (i.e. including the GitHub URL in a requirements.txt file).

Could be worth exploring trying to integrate the installation of Node/npm/Grunt/Gulp/Bower into the setup/frontendbuild.sh file.
@cfpb/front-end-team-admin have we considered this?

@Scotchester @cfpb/front-end-team-admin I think I'd much rather include that node+npm are requirements in the README. Everything else should be able to be pulled in non-globally via npm, right?

@Scotchester @cfpb/front-end-team-admin I think I'd much rather include that node+npm are requirements in the README. Everything else should be able to be pulled in non-globally via npm, right?

This is how we have it in flapjack. Our global requirements are in our INSTALL.md and the rest is in setup.sh. To @Scotchester point, I'm not really clear why some tooling is installed globally. For flapjack this just seems to have evolved that way through the legacy of the build process. I guess the advantage is each project doesn't have to go about installing the base tooling themselves. Also, it's a relatively common workflow to switch branches and run ./setup.sh to clean everything out and get different branches into a clean state, which would be slightly faster with some tools moved to a global context. However, I think the simplification of the installation requirements lead me to +1 to @willbarton idea to make node+npm the only readme requirement and pull everything else in through a build script.

This makes sense to me, having node+npm be our only dependencies.  From an automation perspective it would simplify things, I'd just have to ensure those are in the playbook then pip install vs trying to figure out what each team uses and making sure those are met.

To use Grunt as one example (which may not apply to other programs), grunt-cli can be installed locally, but it's not even discussed as an option in their Getting Started guide, and Node's general guidance on global/local would seem to indicate that global is preferred.
I also found a good argument for not installing globally, particularly when dealing with services like Travis.

nodejs on global vs local guidance

So, the problem seems to be that in order to build front-end dependencies for our projects we need to have front-end build tools (npm, bower, grunt/gulp) installed somehow. But those build tools themselves do not need to travel with the packaged application.
In addition we may have front-end run dependencies that get installed via npm or bower. npm seems like it would be particular problematic in this regard, because we'd have no way to distinguish between build-dependency and run-dependency if they're all installed locally, correct?
@anselmbradford @Scotchester What makes the most amount of sense from a front-end perspective? To require npm/bower/grunt/gulp as required dependencies that we don't provide (like Python)? And then have those tools generate our CSS and JS, as well as install run-dependencies locally?
That puts more dependencies outside the Python installation process, but it might make more sense than pulling everything in. But this could be based entirely on a flawed understanding of all the above tools.

My understanding is that many of our projects would need a global install of Bower and Grunt or Gulp in addition to Node. I just tested this on a clean version of Node and it seems to be the case, even with Grunt declared as a dependency, which unfortunately makes for a much less clean solution.

npm prepends ./node_modules/.bin to the PATH provided to any npm scripts. So if in your package.json file you have:
"scripts": {
  "test": "grunt test",
  "vendor": "bower install"
},
"devDependencies": {
  "grunt": "latest",
  "bower: "latest"
},

It will run A-OK. You don't need to install grunt or bower globally. The user just needs to run npm install beforehand to pull down the deps, which will be stored in ./node_modules and any executables are dumped into ./node_modules/bin.

@contolini But if we do that and need to carry ./npm-modules with us for run-time dependencies (would that be a situation we'd encounter?), that'll bring grunt and bower along for the ride, correct?

@willbarton the run-time dependencies will likely be built into something like bundle.min.js and shipped to the client. So the devDependencies would be excluded.

@willbarton Yeah what @wpears said. There should be no run-time deps stored in ./node_modules. They'll either be consumed by a bundler (like browserify) or they'll be strictly used to build production files.
But if you did copy ./node-modules to a different server, npm would pick them up. Just ensure both environments are running the same version of node.

@contolini @wpears Ok, so really it wouldn't be a problem if we required node/npm, and if bower and grunt aren't available, installed them and anything else locally, and just made sure that ./node_modules doesn't get included with the Python package install, right?

One needs to add grunt-cli as a dependency to use it without the global install. grunt is already a dependency.

@Scotchester Ah, good call. Grunt has a separate cli: https://www.npmjs.com/package/grunt-cli#installing-grunt-cli-locally

Was this comment invisible or something? That's now two reposts of links I posted in that ;)

So it sounds like from the Python setup.py we can call front-end setup.sh file (or frontendbuild.sh, whatever it happens to be) call npm, bower, grunt, and whatever else might be needed to build the front-end, and npm can install the appropriate tools locally if they don't already exist. The end result would be our generated front-end code with dependencies, etc.
At that point, a pip install would result in building the front-end and then discarding all of the locally-installed things. A python setup.py install would not clean up after itself, however, we would also need to provide some customization of the clean task that does that.
Are we okay with this? If so, we could provide our custom setuptools commands in a library that we can just import so projects aren't repeating themselves. I could package up what I've done on reg-site.

@Scotchester What comment? I don't see anything.

@Scotchester

What I think would be best, is if 'setup.py build' could run the front end stuff, and give us the option of having a wheel/egg/zip/sdist/whatever to move around and distribute-- so you could then 'pip install' that package on a machine without needed all of the front-end build dependencies. At that point it's just a plain old python package.

That sounds good too — I took the approach I originally did because we were including the git+https URLs in requirements.txt files. But if we want to move to distributable archives of whatever form (I am a bad Pythonista and have only really glanced at wheels and gone "huh"), that would be fine too.

It leaves the door open, at least: if we don't bother distributing the built packages, it'll still behave the way you describe (because 'install' implies 'build', I think)

:+1:
So I'll package up what I've done here so that we have something reusable, and see if I can do something a little more generic than overriding bdist_egg.

Retirement has a somewhat pressing need to iron out a path to production that unifies Python and front-end builds. We can't update our app because we no longer fit into the current build process.
Could we use Retirement as a test for the @willbarton setup.py scheme? The current blocker is that the built front-end assets don't have a path to cfgov.
This issue and #5 explore having a Python build process that includes front-end concerns; Front-end issue 49 and @anselmbradford suggest having a front-end build that includes Python concerns. The idea of a single script that would build any CFPB project is appealing, but legacy projects don't have front-end pipelines, and shouldn't we also allow for projects that don't include Django or Python? Would it be a nightmare to allow either setup.sh or setup.py to drive the bus? @rosskarchner @Ooblioob

@higs4281 what's the problem exactly? I saw you imported the stuff I did in reg-site... did that solve the problem, or is there something else going on?
I made my changes to reg-site based on our current deployment process, which seems to prefer pip.

The Jenkins build routine for Django apps doesn't include a front-end build process. It assumes front-end assets are in place, and runs collectstatic to round them up. Before we added a front-end build, Retirement fit into this scheme. Now we need a step that would allow us to build resources and put them in place  before collectstatic runs. Node is not currently installed on build, content or prod servers.

Yeah, I'm still not understanding why use setup.py to call setup.sh and not the reverse? setup.sh is meant to be as close to the system and tech-agnostic as possible. I think perhaps since it came up in the front-end repo/evolved out of frontend.sh it's getting confused as being a purely front-end build script, but the idea was something more analogous to a makefile. Maybe having that written in python makes sense for a reason I'm not seeing, but it superficially seems like shell is more appropriate. If we created a nodejs project would it make sense to have a bootscript called setup.js that's the entry point? I don't think so. It would make more sense instead to call the project-specific bootscript from setup.sh.

@anselmbradford Because right now we're installing our projects via pip as eggs or wheels. Doing so presumes that the CSS/JS/HTML/etc travel with the Python package as part of the application in that way.
Without modifying the existing build scripts, that was the easiest way to include the built front-end with the Python package.

@willbarton Ahh okay, so these are python packages that are distributed through a package manager? Is having two installation entry points an option? pip install calls ./setup.sh, but ./setup.sh on its own works comprehensively as well. One for python package distribution and another for manual installation. Perhaps this is the setup you've already written about.

@anselmbradford So, pip allows you to install Python packages as egg files, wheel files, or straight from git. When you install from git directly (i.e. from github) it runs setup.py to build the package. Installing from GitHub is a relatively normal pattern (and allows you to specify requirements that way too) So, if you want to preserve that ability, it has to be setup.py.
I think it makes more conceptual sense that the front-end build stuff happens at the "build" stage of the Python packaging, because it's going to be generating the final CSS/JS that gets served via the Django app.

I will add that how the Python package builds the final front-end code is entirely up to @cfpb/front-end-team-admin. You've settled on a setup.sh file for now for running npm, grunt/gulp, and bower. We could easily invoke them separately as part of the build process too, if needed.

I am not sure if this is till going but I am using https://github.com/nodesource/distributions as a way to install node 0.10. It already has .sh scripts.

I automated it using ansible: https://github.cfpb.gov/ConsumerResponse/complaint-analytics-search-tool-box/tree/master/provisioners/roles/nodejs

Hi! We'll be discussing this issue in today's Dev COP!
I'm asking for two things:

Testable hypotheses: How can we make this work?
Experimentation: Does it work? Why? Can we use it?

Stay tuned for more Project BEFE: Mission: Impawsible

@moka What do you think about trying this on CCDB 4.1?

Here's what I have coming out of our Dev COP meeting last week, where I outlined the problem space. We have two hypotheses at the moment:
@marcesher, what are your thoughts?
Hypotheses:
1. Python Building
Hypothesis
setup.py can integrate the front-end setup.sh and it will be possible to build Python wheels that contain the built front-end assets and the Python code all in one package.
This would result in the most straight-forward deployment of Django app via Python packages, installed via pip into a regular Python location. There would be fewer surprises for those with Django and Python experience.
Testing it
Implement bdist_wheel and bdist_egg in a setup.py file and see if it can successfully invoke setup.sh. Then examine the subsequent wheel file for correct front-end assets.
Does the subsequent Python wheel (or egg) contain the correct and complete front-end assets? Can it be deployed via Jenkins and work?
Potential problems
Only usable with Python back-ends. It doesn't solve any problems we might have integrating the front-end process with other back-ends. This might not be such a bad thing — it might be best to encourage domain-specific solutions rather than a global one-size fits all approach.
Volunteer(s)
@willbarton
2. Tarball (or RPM?)
Hypothesis
We can create a language-agnostic system to build and install single units of software containing components regardless of their implementation languages. Front-end assets can be built and a single tar file (or RPM?) created that can be deployed that contains everything in place for deployment.
Testing it
Implement a simple wrapper (shell script?) around the front-end build process and the Python build process to create the archive.
Does the archive contain all the appropriate front-end and back-end files? Can it be deployed via Jenkins? Does it work? Can it be pluggable, so that we can use it for our Scala and Clojure projects as well?
Potential problems
It would require a re-think in how we deploy software and our dependence on language-based dependency discovery.
Volunteer(s)
???

@willbarton if our standard is Django + Python apps I think an MVP solution should only care about that :+1:

@cfarm I agree to an extent... the only caveat is that a Python-only solution is absolutely not portable to anything else. However, I don't know that we struggle with this problem in any context other than front-end + python.

Just as a note, another alternative is to use Conda instead of pip/PyPi. Conda is meant for packaging Python plus other stuff.

In my testing, this setup.py works.
I'll be presenting in Dev COP tomorrow, but the basic idea is that this setup.py file will create either egg or wheel files (or it can be pip installed directly if npm, gulp, and bower are already installed) which can then be deployed. Meaning that instead of building the front-end artifact, copying it to the deployment target, and then pip installing the Python package, we can generate a single egg or wheel file artifact in the build stage that has the Python packages and the front-end assets, and then pip install that file.
I'll have a more concrete example tomorrow.

:+1: Conda is slick, but if we can adopt your method, I'd vote for avoiding another packaging  dependency.

I've been thinking about this (slowly, obviously...) since Will's presentation, and I'm starting to wonder if this is actually a good path to go down. Django does a pretty good job in allowing your "app", static assets, and even templates to be loosely coupled. They can be in separate locations on the filesystem, and hence can be distinct deployable artifacts.
In other words-- Is this added complexity really just helping us avoid generating two artifacts?  (a python package, and a bundle of static assets). Is that a fair trade?

I think that's a fair trade, as long as we're generating deployable artifacts.
I think the concern is the current process has front-end assets built, then the the Python is pip install -eed from a local checkout, the built front-end assets are copied into that checkout, and then collectstatic is called. That's convoluted.
I could see a process whereby Python wheels are installed, and NPM packages are installed to the Django static location, or something to that effect.

While we await https://github.com/cfpb/drama-free-django with bated breath, why don't we put some effort into creating a sample Jenkins As Code build flow that will build front-end and Python packages and deploy them? Keep with our experimental framework and see how that works?
Does anyone want to take this on?

I don't know much about the JaC part of this, but PR 390 in cfpb_django adds a new "static.in" directory-- we can deposit static assets there, and they will get sucked when collectstatic is run.

Mememememe! I would love to, but will probably need some guidance, so pairing on this with someone is much appreciated in advance.

@rosskarchner :+1: I do like this approach.
@imuchnik I'll happily pair on this.

Is this the idea: All front-end build deployment jobs would push their name-spaced assets to the central static.in directory for collection? Or is this a step toward having all Django apps in the same repo? or both?

It's a place where the front-end deployment jobs can push their namespaced assets.
We'd still want to package Django apps, I think, but this makes one part not dependent on the other. Which seems cleaner and more self-documenting, I think, than some magic in the setup.py file. And, thanks to package_data's obtuseness, it's going to have to be some magic or a lot of explicit paths by hand. (And what could go wrong with that?)


Reduce hero size on iPad  real estate

On an iPad in landscape, too much real estate is consumed vertically. So much so, no data is seen (see attachment).

I'll leave this up to @lukemarohn -- let us know if you think we can use the space more optimally, given that the website is definitely beautiful already


PDT payments intermittently broken?

In some recent requests on my site the query parameters passed to the return_url have changed, in a way that's incompatible with the current implementation of PayPalPDTForm.
Specifically, I'm now receiving:

notify_version=UNVERSIONED which is incompatible with the decimal type of the underlying model field
payment_date=2021-05-11T12:00:00Z which is incompatible with PayPalDateTimeField

Tweaking the form like this works fine locally with Django 3.1 — when I get a moment I'll create a proper patch with a test case.
class PayPalPDTForm(PayPalStandardBaseForm):
    payment_date = DateTimeField(required=False)

    class Meta:
        model = PayPalPDT
        exclude = [
            "ipaddress",
            "flag",
            "flag_code",
            "flag_info",
            "query",
            "response",
            "created_at",
            "updated",
            "form_view",
            "notify_version",
        ]

Edit: I originally thought this was related to v1.1 due to the timing of the issue, but even after I reverted I'm still seeing some requests arrive with the new parameters. Looking at the code I can't see how changing the postback endpoint would cause this issue anyway!

Based on what you've said, it looks like we're going to have to be compatible with all the different things they might be sending. If you can post examples of different things you are seeing that would really help. A patch that fixes the issues would also be a huge help. Thanks so much!

I've been running this branch for the last week or so without any issues, but it's a bit hacky to push upstream:
https://github.com/spookylukey/django-paypal/compare/master...djw:return-query-changes
Given that payment_date is now arriving in ISO format in the query parameters, but still in PayPalDateTimeField format in the postback response, I think the options are:

Extend PayPalDateTimeField to support ISO format (perhaps requiring a dependency on python-dateutil)
Use separate form classes to handle query parameters and the postback response, and probably ignore all the new parameters, including the ISO format date.

I lean towards the second option, but would like your opinion before working on a patch. :)

Your second option sounds fine if you feel like working on a patch. Thanks!

Apologies for the delay, I've created a new branch which implements the fix following Option 2 above:
https://github.com/spookylukey/django-paypal/compare/master...djw:issue-329
I'll test this out for a few days and then submit a PR.


added dataset and task descriptions for batch_integration_graph

Just added some blurb text. Shouldn't need much testing.

I can add the README changes here too... maybe good not to merge yet.

Not quite

Merging so we can post a version of the website without TODOs. We can always continue to review copy later


Merge master to live

Added FAQ about accounts

:x: Validation status: errors

File
Status
Preview URL
Details

:x:Error

Details

Line 16: [Error] Restore nuget package W:\12zs-s.optemp\packages\opbuild.scripts.1.47.0\tools\packages.config in folder:W:\12zs-s.optemp\packages failed, possibly due to transient errors from nuget servers. Please retry building your content. If the issue still happens, open a ticket in http://SiteHelp. Detailed error:  Errors in packages.config projects     Unable to find version '1.47.0' of package 'opbuild.initializecrrtool'.       D:\Windows\system32\config\systemprofile.nuget\packages: Package 'opbuild.initializecrrtool.1.47.0' is not found on source 'D:\Windows\system32\config\systemprofile.nuget\packages'.       https://nuget.org/api/v2/: Package 'opbuild.initializecrrtool.1.47.0' is not found on source 'https://nuget.org/api/v2/'.       https://www.myget.org/F/op/api/v2/: Package 'opbuild.initializecrrtool.1.47.0' is not found on source 'https://www.myget.org/F/op/api/v2/'.       https://www.myget.org/F/docfx/api/v2/: Package 'opbuild.initializecrrtool.1.47.0' is not found on source 'https://www.myget.org/F/docfx/api/v2/'.       https://www.myget.org/F/op-dev/api/v2/: Package 'opbuild.initializecrrtool.1.47.0' is not found on source 'https://www.myget.org/F/op-dev/api/v2/'.       https://www.myget.org/F/docfx-dev/api/v2/: Package 'opbuild.initializecrrtool.1.47.0' is not found on source 'https://www.myget.org/F/docfx-dev/api/v2/'.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Thumbnail Extraction from .mov Issue

Problem
When attempting to extract a thumbnail out of some .mov files recorded via an iphone, the following error is observed:
{
  "level": "warn",
  "ts":<PHONE_NUMBER>.867378,
  "caller": "imagor@v1.4.9/imagor.go:408",
  "msg": "process",
  "params":
    {
      "path": "https%3A%2F%2Fcdn-v2-stage.catchandrelease.com%2Fassets%2Fab866918-65f8-4f52-96d1-ff80532aeaed%2Fa85534be-65e1-46e0-ac63-823a161abca5.mov",
      "image": "https://cdn-v2-stage.catchandrelease.com/assets/ab866918-65f8-4f52-96d1-ff80532aeaed/a85534be-65e1-46e0-ac63-823a161abca5.mov",
      "unsafe": true,
    },
  "error": "imagor: 406 ffmpeg: invalid data found when processing input",
}

Observations
I am able to re-encode the borked .mov with the following command, pass it into imagor via the same flow and observe the extracted thumbnail
ffmpeg -i sample.mov -c:v libx264 -crf 23 -preset veryfast -c:a copy fixed_broken.mov

Questions
disclaimer: i am very ignorant in the realm of video processing

Is there something we should be doing for preprocessing some .mov's before passing them along to imagor?
Could this be a legitmate bug or are we not invoking imagor correctly?

Attachments

failure_01_probe.txt
failure_02_probe.txt
fixed_01_probe.txt

I am so sorry - i opened this in mistake in the wrong repo


Need a way to debug Scala.js offline via sourcemaps

See http://stackoverflow.com/questions/35757352/how-to-debug-scala-js-in-offline-mode

The more recent #2489 is basically a duplicate of this issue, but since there is more discussion on that one, I am closing this issue as duplicate.


Make the User/Client-side Application Golden-Path for Azure.Identity Easy

Tracking issue for addressing UX feedback on User/Client-side Application "Golden path" scenarios with Azure.Identity.  Easy scenarios are still too complicated, require referring to documentation for too many conceptual topics, and it's not clear to new users how best to get started.
This work item tracks adding new simple one-shot or golden-path APIs specific to the User/Client-side Application authentication flows.

Can this be closed now @joshfree?

Closing this because there's nothing in it and it's been superseded by other epics


Generalise job_id retrieval by regex matching

See @willirath PR #131.
I believe this is a really good idea to use regex matching and make it configurable. We should extend this behaviour to other schedulers. Ideally, we would not need to implement _job_id_from_submit_output in JobQueueCluster implementations, and only have it in core.py, retrieving configuration value using scheduler name.

In SGE qsub, there is a command-line argument to just output the job-id (so in SGE there is no need for a regex). Maybe this option exists in other schedulers too?
I am not too sure about making it configurable, I am hoping that we can cover all use cases with a per-scheduler regex that lives in our code rather than saying "if your cluster is a bit weird you can always tweak your dask config".
Ultimately it comes down to how much variation we expect from the submission command output depending on scheduler version or cluster setup. Maybe each cluster defines a qsub wrapper and customise its output? @guillaumeeb you probably have a way better feeling for this than I have so your input on this variability would be more than welcome!

Actually I did not expect variability for PBS, so this is a first example of bad first feeling.
However, with the Torque fork, older PBS versions, there may be some other unexpected outputs.
But my first concern was for LSF, which has a very verbose output that seem to vary according to configuration. And for LSF, I'm almost sure there is no way of outputting only JOB_ID.
Maybe Slurm has some different outputs too, don't know though.
One of the major point of using configurable regex, is to reduce python code by removing all _job_id_from_submit_output methods. So the per-scheduler regex lives in a config, and there is one single line of code.
But maybe this is overkill...

I had a quick look at the infamous ipyparallel file mentioned in #49. You can look for job_id_regexp to figure out which regexp is used by controllers to parse the job-id from the submission command output. The controllers use r'\d+' for finding the job id, except HTCondor which uses r'(\d+)\.$').
Maybe we can just follow what ipyparallel has done and hope that it covers most of schedulers variations out there.
I am still somewhat unsure about making it configurable ... I would still be in favour of not making it configurable, waiting if anyone complains that the regex does not cover their specific cluster and only then make it configurable (an additional line per scheduler in the config).

I had a quick look at the infamous ipyparallel file

Yeah its a tough file !

The controllers all use r'\d+'
Maybe we can just follow what ipyparallel has done

Doing at least that sounds really good! From what I've seen here, this single regex could cover all our implementation needs.
FWIW, here are some example of submission outputs:

PBS

6757651.admin01

Slurm:

sbatch: Submitted batch job 65541

# But with parsable option, it is
65541;cluster

LSF

Job <930> is submitted to default queue <normal>.

HTCondor (taken somewhere on internet):

Submitting job(s)con.
Logging submit event(s).
1 job(s) submitted to cluster 6075.

# but there is the -terse option that should make it like
10744.0 - 10744.2

SGE (with correct option, as I understand):

1234

I am still somewhat unsure about making it configurable

I understand your point, no need to make it configurable if nobody needs it. This is probably true here.
However, in https://github.com/ipython/ipyparallel or https://github.com/jupyterhub/batchspawner, they seem to chose a different approach of making everything configurable if someone ever needs it. Not sure what is the right thing to do. I find it interesting to not being forced to patch or do a PR for every specific case that one can encounter.
Maybe config will be more interesting and cover most edge cases with #133?
For the time being, having only one _job_id_from_submit_output in core.py with a non configurable regex is fine for me.

Closing this as this is resolved in #131


Add missing opamp external outputs for STM32G4

This PR adds the ability to use following opamp/ADC combinations when the opamp is configured in external output mode.
These are documented in the STM32G4 reference manual but currently missing in Embassy.

OPAMP3 with ADC1 (output pin is PB1)
OPAMP4 with ADC1 (output pin is PB12)
OPAMP6 with ADC2 (output pin is PB11)

bender run

Any idea what caused this CI failure?
+ git clone -n https://github.com/G-Levine/embassy.git code Cloning into 'code'... error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) error: 4178 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output run failed: exited with code 128

flaky networking. retrying
bender run


Verilator and icarus verilog produce stack overflow; interpreter produces correct output

As in the title,
fud e --to interpreter-out -s verilog.data DRB1-3123.data DRB1-3123.futil
produces the expected output, while
fud e --to dat --through verilog -s verilog.data DRB1-3123.data DRB1-3123.futil
and
fud e --to dat --through icarus-verilog -s verilog.data DRB1-3123.data DRB1-3123.futil
produce nearly identical error messages:
[fud] ERROR: `/scratch/susan/calyx/target/debug/futil -l /scratch/susan/calyx -b verilog' failed:
---
> [fud] ERROR: `/scratch/susan/calyx/target/debug/futil -l /scratch/susan/calyx -b verilog --disable-init --disable-verify' failed:

The details of verilator's error message is in error.txt. I also uploaded the input files, in case anyone is interested in duplicating the error.
error.txt
DRB1-3123.data.txt
DRB1-3123.futil.txt

Thanks for the bug report! I think it might be guard generator that overflowing so I can take a look. However, we should also chat about the generated code. Everything is sequential so I'm not sure why there are so many groups and invoke statements are generated. The likely cause of the overflow is generating extremely large guards which will result in extremely bad resource usage

Also, @calebmkim, got a new stress test for sharing pass for you:
% ./target/release/futil -b verilog --log info pange.futil > /dev/null
[INFO  calyx::pass_manager] well-formed: 111ms
[INFO  calyx::pass_manager] papercut: 49ms
[INFO  calyx::pass_manager] canonicalize: 55ms
[INFO  calyx::pass_manager] compile-sync: 5ms
[INFO  calyx::pass_manager] group2seq: 28ms
[INFO  calyx::pass_manager] group2invoke: 22ms
[INFO  calyx::pass_manager] inline: 44ms
[INFO  calyx::pass_manager] comb-prop: 43ms
[INFO  calyx::pass_manager] compile-ref: 37ms
[INFO  calyx::pass_manager] infer-share: 3ms
[INFO  calyx::pass_manager] cell-share: 153387ms // <- 2.5 minutes!
[INFO  calyx::pass_manager] remove-comb-groups: 0ms
[INFO  calyx::pass_manager] infer-static-timing: 330ms
[INFO  calyx::pass_manager] compile-invoke: 94ms
[INFO  calyx::pass_manager] merge-static-par: 68ms
[INFO  calyx::pass_manager] static-par-conv: 1630ms
[INFO  calyx::pass_manager] dead-group-removal: 40ms
[INFO  calyx::pass_manager] collapse-control: 16ms
[INFO  calyx::pass_manager] tdcc: 6399ms
[INFO  calyx::pass_manager] dead-group-removal: 11ms
[INFO  calyx::pass_manager] comb-prop: 9997ms
[INFO  calyx::pass_manager] dead-cell-removal: 9044ms
[INFO  calyx::pass_manager] go-insertion: 33ms
[INFO  calyx::pass_manager] wire-inliner: 1789ms
[INFO  calyx::pass_manager] clk-insertion: 66ms
[INFO  calyx::pass_manager] reset-insertion: 54ms
[INFO  calyx::pass_manager] merge-assigns: 602ms

Okay, according to @susan-garry, y'all previously had programs with the same number of PEs and memories and the only change is generating invoke statements which is curious. Some things to note:

This is probably the largest Calyx program the compiler has encountered! Congratulations!
You probably want run the compiler in release mode so its much faster. You can do this by building using cargo build --release and changing fud to use the release compiler (fud c futil.exec "$pwd/target/release/futil" in the Calyx repo)

Now onto sources of overflow:

By running the compiler with --log info (fud e -s futil.flags ' -p all --log info ...), you get the above log which shows none of the compiler passes overflow which is good
The problem is in the backend with the likely culprit being this recursive function: https://github.com/cucapra/calyx/blob/master/src/backend/verilog.rs#L483
The other, much worse possibility, is that the pretty printer in the Verilog library we use cannot print out the Verilog AST we're generating. This would be much harder to fix unfortunately

To get the overflow quicker, we can run the compiler with optimizations disabled using -p no-opt:
% ./target/release/futil -b verilog --log info -p no-opt pange.futil > /dev/null
[INFO  calyx::pass_manager] well-formed: 98ms
[INFO  calyx::pass_manager] papercut: 48ms
[INFO  calyx::pass_manager] canonicalize: 65ms
[INFO  calyx::pass_manager] compile-sync: 5ms
[INFO  calyx::pass_manager] compile-ref: 41ms
[INFO  calyx::pass_manager] remove-comb-groups: 0ms
[INFO  calyx::pass_manager] compile-invoke: 86ms
[INFO  calyx::pass_manager] tdcc: 6388ms
[INFO  calyx::pass_manager] go-insertion: 61ms
[INFO  calyx::pass_manager] wire-inliner: 2021ms
[INFO  calyx::pass_manager] clk-insertion: 93ms
[INFO  calyx::pass_manager] reset-insertion: 92ms
[INFO  calyx::pass_manager] merge-assigns: 979ms

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
zsh: abort      ./target/release/futil -b verilog --log info -p no-opt pange.futil > /dev/nul

Running the compiler to just print out the Calyx program after compilation doesn't overflow:
% ./target/release/futil --log info -p no-opt pange.futil > out.futil
[INFO  calyx::pass_manager] well-formed: 111ms
[INFO  calyx::pass_manager] papercut: 47ms
[INFO  calyx::pass_manager] canonicalize: 118ms
[INFO  calyx::pass_manager] compile-sync: 13ms
[INFO  calyx::pass_manager] compile-ref: 42ms
[INFO  calyx::pass_manager] remove-comb-groups: 1ms
[INFO  calyx::pass_manager] compile-invoke: 89ms
[INFO  calyx::pass_manager] tdcc: 6474ms
[INFO  calyx::pass_manager] go-insertion: 58ms
[INFO  calyx::pass_manager] wire-inliner: 2067ms
[INFO  calyx::pass_manager] clk-insertion: 83ms
[INFO  calyx::pass_manager] reset-insertion: 87ms
[INFO  calyx::pass_manager] merge-assigns: 957ms

Running with lldb, I get this:
% lldb -- ./target/release/futil -b verilog --log info -p no-opt pange.futil
...
Process 69743 stopped
* thread #1, name = 'main', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x7ff7bf700ff8)
    frame #0: 0x0000000100295824 futil`core::ptr::drop_in_place$LT$pretty..Doc$LT$pretty..RcDoc$GT$$GT$::hc5ff1d9bc7dd2c56 + 132
futil`core::ptr::drop_in_place$LT$pretty..Doc$LT$pretty..RcDoc$GT$$GT$::hc5ff1d9bc7dd2c56:
->  0x100295824 <+132>: callq  0x1002957a0               ; <+0>
    0x100295829 <+137>: movq   0x8(%rbx), %rdi
    0x10029582d <+141>: callq  0x1002d6778               ; symbol stub for: free
    0x100295832 <+146>: movq   0x10(%rbx), %rdi
Target 0: (futil) stopped.
(lldb)

RcDoc is the library vast, our verilog backend, uses to print out verilog strings. The EXC_BAD_ACCESS is concerning because it seems to imply a bad memory access which should not happen in Rust? RcDoc probably uses unsafe code which might be misbehaving? Still, the fact what we're getting to pretty printing means that its probably something wrong in vast or RcDoc.
Also, summoning @EclecticGriffin's Rust powers in case they have better ideas of what could be going wrong

Building with AddressSanitizer:
AddressSanitizer:DEADLYSIGNAL
=================================================================
==72654==ERROR: AddressSanitizer: stack-overflow on address 0x7ff7b67509b8 (pc 0x000109f797df bp 0x7ff7b67511f0 sp 0x7ff7b67509c0 T0)
    #0 0x109f797df in __asan_memcpy+0x18f (librustc-nightly_rt.asan.dylib:x86_64+0x467df) (BuildId: 364f88c707f13921b85413add38b47d4240000001000000000070a0000010c00)
    #1 0x1097cf2b8 in arrayvec::array_string::ArrayString$LT$A$GT$::new::he839dde011b09996+0xc8 (futil:x86_64+0x10081f2b8) (BuildId: 8c58d4e6d80a312cb8d786fa62c5165932000000200000000100000000000d00)
    #2 0x1097ad951 in pretty::RcDoc$LT$A$GT$::as_string::h7398701d6df83bc0+0x121 (futil:x86_64+0x1007fd951) (BuildId: 8c58d4e6d80a312cb8d786fa62c5165932000000200000000100000000000d00)
    #3 0x1097b50b2 in vast::subset::pretty_print::_$LT$impl$u20$vast..util..pretty_print..PrettyPrint$u20$for$u20$vast..subset..ast..Expr$GT$::to_doc::h91b5fcf4d36ccf10+0x11d2 (futil:x86_64+0x1008050b2) (BuildId: 8c58d4e6d80a312cb8d786fa62c5165932000000200000000100000000000d00)
    #4 0x1097b5ac1 in vast::subset::pretty_print::_$LT$impl$u20$vast..util..pretty_print..PrettyPrint$u20$for$u20$vast..subset..ast..Expr$GT$::to_doc::h91b5fcf4d36ccf10+0x1be1 (futil:x86_64+0x100805ac1) (BuildId: 8c58d4e6d80a312cb8d786fa62c5165932000000200000000100000000000d00)
    #5 0x1097b624c in vast::subset::pretty_print::_$LT$impl$u20$vast..util..pretty_print..PrettyPrint$u20$for$u20$vast..subset..ast..Expr$GT$::to_doc::h91b5fcf4d36ccf10+0x236c (futil:x86_64+0x10080624c) (BuildId: 8c58d4e6d80a312cb8d786fa62c5165932000000200000000100000000000d00)
    #6 0x1097b624c in vast::subset::pretty_print::_$LT$impl$u20$vast..util..pretty_print..PrettyPrint$u20$for$u20$vast..subset..ast..Expr$GT$::to_doc::h91b5fcf4d36ccf10+0x236c (futil:x86_64+0x10080624c) (BuildId: 8c58d4e6d80a312cb8d786fa62c5165932000000200000000100000000000d00)
    #7 0x1097b624c in vast::subset::pretty_print::_$LT$impl$u20$vast..util..pretty_print..PrettyPrint$u20$for$u20$vast..subset..ast..Expr$GT$::to_doc::h91b5fcf4d36ccf10+0x236c (futil:x86_64+0x10080624c) (BuildId: 8c58d4e6d80a312cb8d786fa62c5165932000000200000000100000000000d00)
    #8 0x1097b624c in vast::subset::pretty_print::_$LT$impl$u20$vast..util..pretty_print..PrettyPrint$u20$for$u20$vast..subset..ast..Expr$GT$::to_doc::h91b5fcf4d36ccf10+0x236c (futil:x86_64+0x10080624c) (BuildId: 8c58d4e6d80a312cb8d786fa62c5165932000000200000000100000000000d00)
...
  #254 0x1097b624c in vast::subset::pretty_print::_$LT$impl$u20$vast..util..pretty_print..PrettyPrint$u20$for$u20$vast..subset..ast..Expr$GT$::to_doc::h91b5fcf4d36ccf10+0x236c (futil:x86_64+0x10080624c) (BuildId: 8c58d4e6d80a312cb8d786fa62c5165932000000200000000100000000000d00)

Looks like a bug in the pretty printing implementation in vast?

Okay, I've reduced the test case and it seems to come from...the cells? You don't even the control program. Just a lot of cells

Nice work isolating the "big cells list" test case, @rachitnigam! Given that it's a segfault, I share the intuition that it could just be a stack overflow in VAST… I'll see if I have a moment to break out lldb as well.

Just to add on to this:
I am getting an identical error when trying to get resource estimates when I fully inline some of the larger Calyx neural networks.  Based on this thread, I'm guessing that when I fully inline everything, it's adding a bunch of new cells to main causing an overflow when we try to go from compiled Calyx -> Verilog.

The interesting thing is that removing the external attribute from cells makes the compilation work. I’d have to see if that’s really something or just a consequence of making the generated code smaller

Okay, I've debugged this some more and the problem is pretty wild–VAST generates really large modules which the pretty printing module turns into string. Everything is fine till the pretty printing function goes to return the value. The function runs drop to drop the struct allocated by VAST's representation of the documentation and blows the stack during the drop process

That is absolutely bonkers 👏

I tried running fud e --to dat --through verilog -s verilog.data DRB1.data DRB1.futil (after updating calyx), but I still get a similar overflow error. The solution seems to be to rewrite the program to be smaller, but I want to check that I'm not missing something since @rachitnigam mentioned being able to compile this program to verilog in #1280.

@susan-garry are you building and running the compiler in release mode. You need to build the compiler using release mode:
cargo build --release

And then change fud to use the release binary:
fud e stages.futil.exec "<calyx repo>/target/release/futil

It should not get a overflow error anymore. If it does, open a new issue please


unable to start chrome using existing session

unable to start chrome using existing session even i comment out "-incognito" in launch.json. Chrome will start in clean, i need to re-login to sharepoint. How to fix it?
thanks

Thanks for you interest! However, this is an issue with Chrome, not SharePoint development so it will be closed for now. If you are convinced the issue is with the SPFx toolchain, please provide additional details such as the version of SPFx and the versions of your other tools (node, chrome, etc.). Thanks!


remove changelog as we use autogenerated changes

Codecov Report

Merging #45 (7227820) into main (112933a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #45   +/-   ##
=======================================
  Coverage   82.49%   82.49%
=======================================
  Files          13       13
  Lines         497      497
=======================================
  Hits          410      410
  Misses         87       87

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Update type definitions

Apparently ItemSeparatorComponent isn't supported.

I too have been having this problem. Also I do not understand where the TypeScript definitions should be coming from. Does this library support them, or do you need a @types library still?

types have been updated


Allow editting the map mode sprite of a planet that's already been loaded

What Happened?
Changing another NH config's ShipLog.mapMode fields does not properly get applied. Currently tested with revealedSprite and invisibleWhenHidden. I have a mod that is patching another mod and adds these fields, and those additions were ignored.
What was supposed to happen?
These fields should be merged/added to the base config at runtime.
Platform
Steam
Mods
xen.NewHorizons
CrypticBird.Jam3

Logs
No response

this doesnt happen because nh does not merge planet configs at all
contrasting to systems, with are merged into one config and then applied, planet configs are just applied sequentially with no regards to overwrites (because most properties are additive in nature ("add this detail") and not updating a value)

We could instead generally allow updating the map mode icons of existing planets. This would include modded and stock.
In UpdateBody we'd need to find the ShipLogAstroObject that corresponds to the existing planet. Then take its _unviewedObj, _imageObj, and _outlineObj game objects and replace their Image component's textures.
Better yet we just delete those objects entirely and call some of the code from MapModeBuilder.AddShipLogAstroObject and replace them. Or something!

actually it appears to just Instantiate a new thing for _unviewedObj and just uhhhh forget about (but not delete) whatever existed there before. so if multiple planets go make then i think it would just overwrite but keep the one ones disabled(?) but existing forever


KAFKA-6181 Examining log messages with {{--deep-iteration}} should show superset of fields

adding missing fields for --deep-iteration log messages.
Missing fields include : partitionLeaderEpoch, baseSequence, lastSequence, etc.,

please review this @mjsax

Hi @chia7712  @hachikuji @cmccabe please review this. Thanks.

Thanks for approving @chia7712 , have attached the images of before and after message.

@iprithv Thanks for the sharing. nit: there is a redundant space before “batchSize”.

@chia7712 have removed the redundant space and have attached a screenshot. Please do review and merge the same.

@iprithv Could you rebase code (or add trivial change) to trigger QA again?

rebase patch to trigger QA

@iprithv Could you share your jira account to me? I want to assign https://issues.apache.org/jira/browse/KAFKA-6181 to you.

Sincere apologies for the delay in response. My jira account is<EMAIL_ADDRESS>Username: iprithv
Full name: Prithvi
Also I have commented in Jira.

@iprithv Thanks for you patch. I have assigned https://issues.apache.org/jira/browse/KAFKA-6181 to you :)