Self-referential one-to-many relationship query fails: "subquery has too many columns"

Bug description
I have a User model with a one-to-many self-referential relationship, as follows:
model User {
  id         Int     @id @default(autoincrement())
  name       String?
  friends    User[]  @relation("UserfriendOf")
  friendOf   User?   @relation("UserfriendOf", fields: [friendOfId], references: [id])
  friendOfId Int?
}

A user can be the friendOf one person, but can be friends with many (forgive the contrived example).
I want to filter the users based on properties of the friendOf field, for example:
  await prisma.user.findMany({where: { friendOf: { name: { contains: "B" } } } });

Running this query causes the prisma engine to throw the following error:
Invalid `prisma.user.findMany()` invocation in
/Users/timleslie/src/prisma-issue/index.ts:6:21

  2 const prisma = new PrismaClient()
  3
  4 async function main() {
  5   // Filter based on the name of the friend
→ 6   await prisma.user.findMany(
  Error occurred during query execution:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState("42601"), message: "subquery has too many columns", detail: None, hint: None, position: Some(Original(163)), where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("parse_expr.c"), line: Some(2070), routine: Some("transformSubLink") }) }) })
    at PrismaClientFetcher._request (/Users/timleslie/src/prisma-issue/node_modules/@prisma/client/src/runtime/getPrismaClient.ts:999:15)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
(node:75365) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:75365) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

A full debug run gives:
  prisma-client {
  prisma-client   engineConfig: {
  prisma-client     cwd: '/Users/timleslie/src/prisma-issue/prisma',
  prisma-client     debug: false,
  prisma-client     datamodelPath: '/Users/timleslie/src/prisma-issue/node_modules/.prisma/client/schema.prisma',
  prisma-client     prismaPath: undefined,
  prisma-client     engineEndpoint: undefined,
  prisma-client     generator: {
  prisma-client       name: 'client',
  prisma-client       provider: 'prisma-client-js',
  prisma-client       output: '/Users/timleslie/src/prisma-issue/node_modules/@prisma/client',
  prisma-client       binaryTargets: [],
  prisma-client       experimentalFeatures: [Array],
  prisma-client       config: {}
  prisma-client     },
  prisma-client     showColors: false,
  prisma-client     logLevel: undefined,
  prisma-client     logQueries: undefined,
  prisma-client     flags: [],
  prisma-client     clientVersion: '2.2.2',
  prisma-client     enableExperimental: [ 'aggregations' ]
  prisma-client   }
  prisma-client } +0ms
  prisma-client Prisma Client call: +108ms
  prisma-client prisma.user.findMany({
  prisma-client   where: {
  prisma-client     friendOf: {
  prisma-client       name: {
  prisma-client         contains: 'B'
  prisma-client       }
  prisma-client     }
  prisma-client   }
  prisma-client }) +1ms
  prisma-client Generated request: +0ms
  prisma-client query {
  prisma-client   findManyUser(where: {
  prisma-client     friendOf: {
  prisma-client       name_contains: "B"
  prisma-client     }
  prisma-client   }) {
  prisma-client     id
  prisma-client     name
  prisma-client     friendOfId
  prisma-client   }
  prisma-client }
  prisma-client  +0ms
  engine { cwd: '/Users/timleslie/src/prisma-issue/prisma' } +0ms
  engine {
  engine   dotPrismaPath: '/Users/timleslie/src/prisma-issue/node_modules/.prisma/client/query-engine-darwin'
  engine } +0ms
  plusX Execution permissions of /Users/timleslie/src/prisma-issue/node_modules/.prisma/client/query-engine-darwin are fine +0ms
  engine {
  engine   flags: [ '--enable-experimental=aggregations', '--enable-raw-queries' ]
  engine } +1ms
  engine port: 65291 +3ms
  engine stdout {
  timestamp: 'Jul 20 09:03:17.028',
  level: 'INFO',
  target: 'quaint::pooled',
  fields: { message: 'Starting a postgresql pool with up to 9 connections.' }
} +22ms
  engine stdout {
  timestamp: 'Jul 20 09:03:17.035',
  level: 'INFO',
  target: 'query_engine::server',
  fields: { message: 'Started http server on <IP_ADDRESS>:65291' }
} +6ms
  prisma-client Client Version 2.2.2 +37ms
  prisma-client Engine Version a9e8c3d97ef2a0cf59256e6b26097f2a80f0a6a4 +0ms
  engine {
  engine   error: PrismaClientUnknownRequestError: Error occurred during query execution:
  engine   ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState("42601"), message: "subquery has too many columns", detail: None, hint: None, position: Some(Original(163)), where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("parse_expr.c"), line: Some(2070), routine: Some("transformSubLink") }) }) })
  engine       at NodeEngine.graphQLToJSError (/Users/timleslie/src/prisma-issue/node_modules/@prisma/engine-core/dist/NodeEngine.js:761:1)
  engine       at /Users/timleslie/src/prisma-issue/node_modules/@prisma/engine-core/dist/NodeEngine.js:709:1
  engine       at process.result (internal/process/task_queues.js:93:5)
  engine       at PrismaClientFetcher._request (/Users/timleslie/src/prisma-issue/node_modules/@prisma/client/src/runtime/getPrismaClient.ts:957:15)
  engine } +37ms
  prisma-client Error: Error occurred during query execution:
  prisma-client ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState("42601"), message: "subquery has too many columns", detail: None, hint: None, position: Some(Original(163)), where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("parse_expr.c"), line: Some(2070), routine: Some("transformSubLink") }) }) })
  prisma-client     at NodeEngine.graphQLToJSError (/Users/timleslie/src/prisma-issue/node_modules/@prisma/engine-core/dist/NodeEngine.js:761:1)
  prisma-client     at /Users/timleslie/src/prisma-issue/node_modules/@prisma/engine-core/dist/NodeEngine.js:709:1
  prisma-client     at process.result (internal/process/task_queues.js:93:5)
  prisma-client     at PrismaClientFetcher._request (/Users/timleslie/src/prisma-issue/node_modules/@prisma/client/src/runtime/getPrismaClient.ts:957:15) +35ms
  engine Stopping Prisma engine +4ms
(node:75376) UnhandledPromiseRejectionWarning: Error:
Invalid `prisma.user.findMany()` invocation in
/Users/timleslie/src/prisma-issue/index.ts:6:21

  2 const prisma = new PrismaClient()
  3
  4 async function main() {
  5   // Filter based on the name of the friend
→ 6   await prisma.user.findMany(
  Error occurred during query execution:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState("42601"), message: "subquery has too many columns", detail: None, hint: None, position: Some(Original(163)), where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("parse_expr.c"), line: Some(2070), routine: Some("transformSubLink") }) }) })
    at PrismaClientFetcher._request (/Users/timleslie/src/prisma-issue/node_modules/@prisma/client/src/runtime/getPrismaClient.ts:999:15)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
(node:75376) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:75376) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

How to reproduce

Create a new project as per https://www.prisma.io/docs/getting-started/setup-prisma/start-from-scratch-prisma-migrate-typescript-postgres
Edit the schema.prisma file with the schema given below.
Run the migrations and generate the client as per the docs
Create an index.ts file with the contents given below
Run DEBUG="*" npx ts-node index.ts

Expected behavior
The findMany query should not throw an exception.
Prisma information
schema.prisma
datasource postgresql {
  url      = env("DATABASE_URL")
  provider = "postgresql"
}

generator client {
  provider = "prisma-client-js"
}

model User {
  id         Int     @id @default(autoincrement())
  name       String?
  friends    User[]  @relation("UserfriendOf")
  friendOf   User?   @relation("UserfriendOf", fields: [friendOfId], references: [id])
  friendOfId Int?
}

index.ts
import { PrismaClient } from '@prisma/client'
const prisma = new PrismaClient()

async function main() {
  // Filter based on the name of the friend
  await prisma.user.findMany({where: { friendOf: { name: {contains: "B" } } } });
}

main()
  .catch((e) => {
    throw e
  })
  .finally(async () => {
    await prisma.disconnect()
  })

Environment & setup

OS: Mac OS 10.15.5
Database: PostgreSQL 12
Prisma version: 2.2.2
Node.js version: 12.11.0

I am able to reproduce this with 2.4.0-dev.15

Confirming that this is still an issue in 2.4.0 👍

This is most probably also still an issue with 2.5.0 - but we are starting work on investigating and potentially fixing this soon.

Thanks for the feedback @janpio, much appreciated 🙏

Can confirm that the issue I was seeing has been fixed 👍


Clean up Rmd files

[x] read_data.Rmd: @LienReyserhove script to use to read from GS, match with GBIF and write to interim
[x] dwc_mapping.Rmd: @DimEvil basis for full DwC mapping from interim files.

Remove any name match functionality
Read interim EU taxa => taxon core + EU distribution
Read interim regional taxa => link to taxon core using EU name + add distribution rows for the countries

[x] weighted_red_list_values.Rmd: @LienReyserhove file to use for @dirkmaes33 code
[x] Species Red list Butterflies Locale RMD.Rmd: @LienReyserhove move name match functionality to read_data.Rmd and then remove script
[ ] convert_to_dwc.R: legacy script, remove eventually
[x] template Rmd.Rmd: remove script

Decided with @DimEvil that there is no need to match names with GBIF as we don't need any extra information from GBIF. That means that the read step becomes very simple and could as well be integrated in dwc_mapping.Rmd. For now, we have kept it as a separate Rmd.

We do need to change the readme.md file, according to the latest workflow.

Attempt to adapt readme.md


Test suite failure in 70-test_sslrecords.t

In https://buildd.debian.org/status/fetch.php?pkg=openssl&arch=kfreebsd-i386&ver=1.1.0f-1&stamp=1495748849&raw=0 I'm getting:
Test Summary Report
-------------------
../../test/recipes/70-test_sslrecords.t     (Wstat: 0 Tests: 10 Failed: 0)
  Parse errors: Tests out of sequence.  Found (5) but expected (4)
                Tests out of sequence.  Found (6) but expected (5)
                Tests out of sequence.  Found (7) but expected (6)
                Tests out of sequence.  Found (8) but expected (7)
                Tests out of sequence.  Found (9) but expected (8)
Displayed the first 5 of 8 TAP syntax errors.
Re-run prove with the -p option to see them all.
Files=95, Tests=534, 71 wallclock secs ( 1.27 usr  0.37 sys + 52.25 cusr 19.22 csys = 73.10 CPU)
Result: FAIL

I have seen this in Travis too:
https://travis-ci.org/openssl/openssl/jobs/234814816
It doesn't seem to be happening at the moment, so I think it is intermittent. So far I have not been able to reproduce it.

Here I see more of the same type, also in other tests:
https://buildd.debian.org/status/fetch.php?pkg=openssl&arch=hurd-i386&ver=1.1.0f-1&stamp=1495750462&raw=0

But can you reproduce locally? If so...how?

The only thing I can say currently is that I only see it in shared builds. Since that's now the default, I guess you test that too?

Yes

It at least seems very reproducible on hurd-i386:
https://buildd.debian.org/status/logs.php?pkg=openssl&ver=1.1.0f-1&arch=hurd-i386

I think I know what is happening. There is a race condition going on. I think the bug has been there for a long time, but I suspect commit 76e0d0b21c has somehow made it visible because it is related to the way the test output is printed and parsed by the harness.
I took a detailed look at the output from this travis failure:
https://travis-ci.org/openssl/openssl/jobs/236307668
The complaint from the test harness is this:
../test/recipes/70-test_sslrecords.t     (Wstat: 0 Tests: 10 Failed: 0)
  Parse errors: Tests out of sequence.  Found (4) but expected (3)
                Tests out of sequence.  Found (5) but expected (4)
                Tests out of sequence.  Found (6) but expected (5)
                Tests out of sequence.  Found (7) but expected (6)
                Tests out of sequence.  Found (8) but expected (7)
Displayed the first 5 of 9 TAP syntax errors.

IIRC the test harness parses the output from the tests looking for the "ok" line but, from the above errors, it looks like it sees tests 1 and 2, and then the next test result it sees is 4! So what happens to test 3? Well it is run and the output is there. Here it is:
Waiting for server process to close: 22624
depth=0 C = UK, O = OpenSSL Group, OU = FOR TESTING PURPOSES ONLY, CN = Test Server Cert
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = UK, O = OpenSSok 3 - Too many in context empty records test
L Group, OU = FOR TESTING PURPOSES ONLY, CN = Test Server Cert
verify error:num=21:unable to verify the first certificate
verify return:1

So, the test harness it printing out the results of the test, but some other process is still running at the time and it is printing out stuff, so the "ok" message appears in the middle of a line where the test harness is not expecting to find it.
Looking at the proxy, it closes off a test run like this:
    if($server_sock) {
        $server_sock->close();
    }
    if($client_sock) {
        #Closing this also kills the child process
        $client_sock->close();
    }
    if($proxy_sock) {
        $proxy_sock->close();
    }
    if(!$self->debug) {
        select($oldstdout);
    }
    $self->serverconnects($self->serverconnects - 1);
    if ($self->serverconnects == 0) {
        die "serverpid is zero\n" if $self->serverpid == 0;
        print "Waiting for server process to close: "
              .$self->serverpid."\n";
        waitpid( $self->serverpid, 0);
        die "exit code $? from server process\n" if $? != 0;
    }
    return 1;

So you can see it waits for the server pid to indicate the process has finished before continuing, but on the client side it just relies on the client to close itself down when the proxy closes its socket. If the client is busy doing something else at the time (like printing stuff on the screen), then it may not notice immediately so there may be a short delay until it closes itself down. Probably what we need to do is make the Proxy wait on the client process pid too.

See #3567 for my proposed fix.


feat: Conditionally allow to keep partition_by columns when using PARTITIONED BY enhancement

Which issue does this PR close?
Closes #10971
What changes are included in this PR?

Added a flag to FileSinkConfig to conditionally enable this feature. Disabled by default.

Are these changes tested?
Test is included as part of copy.slt.
Are there any user-facing changes?
No breaking changes.
Added some documentation for this new option.

One last thing we could add in this PR (or a ticket for a follow on) would be to add a session level configuration setting for this, so users could control the default behavior.

I think I have address all the comments in the PR, the only lingering one is about flowing the new option from SQL when using CREATE EXTERNAL TABLE. Thank you for the quick turnaround on the reviews and comments.

Thank you for the updates @hveiga ! The new session parameter is a great addition. I do think we should avoid changing the CopyTo struct.
Updates to create external table can wait for a future PR in my view.

Thanks for the review. I reverted the hive_options changes and the PR is ready for (yet another) review :)

I took the liberty of pushing some commits to this PR to fix some CI errors

I merged up from main to resolve some conflicts (and also now so that CI will run automatically). Once CI passes I plan to merge this PR

Thanks again @hveiga  and @berkaysynnada

Thanks everyone for the help to get this over the finish line! Excited to see this in the next release of Datafusion 🎉

🚀


Hover text is broken

Post-merge
Hovering over functions or other T-SQL text causes a tooltip that just says "Loading..." to appear and the console is filled with these messages :
mainThreadExtensionService.ts:80 Error: Message header must separate key and value using :
at c:\src\azuredatastudio\extensions\mssql\node_modules\vscode-jsonrpc\lib\messageReader.js:57:23
at Array.forEach (<anonymous>)
at MessageBuffer.tryReadHeaders (c:\src\azuredatastudio\extensions\mssql\node_modules\vscode-jsonrpc\lib\messageReader.js:54:17)

Closing this - it's due to the latest sqlparser I'm using locally so not an issue in ADS currently.


AME still glows and bonks after running out of fuel

Description
After an AME has used the last fuel in a bottle, or is started with an empty bottle inserted, the cores still glow and it still makes the injection sound.
It should somehow indicate visually or audio wise if it ran out.
Reproduction

build an AME
insert a fuel jar
wait until it is empty

Screenshots

Additional context
For example, if it tries to inject fuel but can't due to there being no fuel left, it can just disable injections.

this is pretty confusing for new players
the bonk sound should be quieter when theres no fuel, like how its louder when being blown up

im trying to figure this out, but i cant find the ame bonk audio

InjectSound on AmeControllerComponent

the bonk sound should be quieter when theres no fuel, like how its louder when being blown up

Why would it even make a sound after having no fuel left to inject?
My suggestion is to just shut down the AME when it tries to inject fuel but has nothing left:

At least in my headcanon, the noise is the antimatter reacting inside the reactor. Too much antimatter - louder noise. No antimatter - no noise
Programming-wise it would be really easy to implement since it just uses existing states of the AME
Gamebalance-wise it could be argued if the AME running empty should be made more subtle than just being "off" or how clearly we want to communicate to players when the AME runs out. In my opinion, having no noise is actually the more subtle choice since a player with some experience can instantly recognize the AME state from the noise, but having no noise will only attract attention after some time.

ok yeah, maybe no noise for no fuel, quieter when underinjecting so anyone can tell when bozos doing 4 core 4 injection

Closed by #20458


Add pre commit hook for license headers

Was introduced with https://github.com/DSD-DBS/capella-collab-manager/commit/e2b2bb927d1b4c71748301372fc23a520d6153d5


Added note about service communication API retirement as per PM

Docs Build status updates of commit 363e2ea:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

office-365-management-api/office-365-service-communications-api-reference.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Test ign-cmake2 revert

Test that https://github.com/ignitionrobotics/ign-cmake/pull/120 works correctly
Signed-off-by: Michael Carroll<EMAIL_ADDRESS>

This can be closed once ignitionrobotics/ign-cmake#120 lands.


Perf regression: pipenv lock creates many full-tree copies of the project

Issue description
While trying to debug why pipenv update was taking a monumental amount of time, I noticed that the pipenv/resolver.py process was very busy with copying across data to /tmp/reqlib-src[randomvalue] directories. The project includes several GBs of log files, so this was eating up a lot of /tmp disk space and taking a lot of time.
I traced this to the following SetupInfo.from_ireq() lines:
https://github.com/pypa/pipenv/blob/b29a4884e24dd2dd72f379d64b23779a676d2ef9/pipenv/vendor/requirementslib/models/setup_info.py#L1877-L1880
This creates multiple copies of the source tree (in apparent defiance of the LRU decorator), even though the project is listed just once, with no other entries under [packages]:
[packages]
project-name = {editable = true,path = "."}

It appears to create a copy per dependency; we have 104:
$ pipenv graph --json | grep '"key"' | cut -d':' -f2 | sort -u | wc -l
104

and while the temp dirs get cleaned up once the process finishes, towards the end I counted nearly as many temp directories:
$ ls -1ld /tmp/reqlib-src*/project-name | wc -l
85

I'm assuming at this point that had I put in a breakpoint somewhere I'd have seen 104 directories before pipenv completes. (addendum: my later timing tests show that this isn't quite the case, but still close).
While having log files in a project source directory is not ideal, it was not otherwise a problem as setuptools, .gitignore and MANIFEST.in files are configured to ignore the log files. We can't work around the issue with a symlink either as shutil.copytree() is called with symlinks=True.
Even then, nearly 100 copies of the full project is a huge waste of resources.
Expected result
If pipenv must create an isolated environment, it should either attempt to enumerate the source distribution files to copy, or warn or clearly document that a full copy is created of the whole tree. It should then create just one copy.
By temporarily removing the log files, pipenv update completed (albeit still slowly) without completely trashing the filesystem.
Actual result
Either pipenv update times out (in pexpect), or you run out of disk space on your temp partition.

Note: I've cut sensitive information out of the --support output; the project dependencies are simply:
install_requires =
    apache-airflow[celery,postgres,redis] >= 1.10.11
    airflow_multi_dagrun
    airflow-prometheus-exporter

$ pipenv --support
Pipenv version: '2020.6.2'
Pipenv location: '/home/ubuntu/.local/lib/python3.6/site-packages/pipenv'
Python location: '/home/ubuntu/miniconda3/envs/project-name/bin/python'
Python installations found:

3.7.4: /home/ubuntu/miniconda3/bin/python3.7m
3.7.4: /home/ubuntu/miniconda3/bin/python3
3.7.4: /home/ubuntu/miniconda3/bin/python3.7
3.6.9: /usr/bin/python3
3.6.9: /usr/bin/python3.6m
3.6.9: /usr/bin/python3.6
2.7.17: /usr/bin/python2
2.7.17: /usr/bin/python2.7

PEP 508 Information:
{'implementation_name': 'cpython',
 'implementation_version': '3.6.10',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.3.0-1027',
 'platform_system': 'Linux',
 'platform_version': '#29~18.04.1-Ubuntu SMP Mon Jun 22 15:19:42 UTC 2020',
 'python_full_version': '3.6.10',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

CONDA_SHLVL
LC_ALL
LS_COLORS
LD_LIBRARY_PATH
CONDA_EXE
SSH_CONNECTION
LESSCLOSE
LANG
CONDA_PREFIX
S_COLORS
_CE_M
XDG_SESSION_ID
USER
PWD
HOME
CONDA_PYTHON_EXE
LC_CTYPE
LC_TERMINAL
SSH_CLIENT
TMUX
LC_TERMINAL_VERSION
XDG_DATA_DIRS
COMBILEXDIR
_CE_CONDA
LADSPA_PATH
CONDA_PROMPT_MODIFIER
SSH_TTY
MAIL
TERM
SHELL
TMUX_PANE
SHLVL
LOGNAME
XDG_RUNTIME_DIR
PATH
CONDA_DEFAULT_ENV
LESSOPEN
_
OLDPWD
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:
Debug–specific environment variables:

PATH: /home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/miniconda3/bin:/home/ubuntu/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL: /bin/bash
LANG: C.UTF-8
PWD: /home/ubuntu/project-name

Contents of Pipfile ('/home/ubuntu/project-name/Pipfile'):
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
flake8 = "*"
flake8-bugbear = "*"
black = "==19.10b0"
pre-commit = "*"
pytest = "*"
pytest-cov = "*"

[packages]
project-name = {editable = true,path = "."}

[requires]
python_version = "3.6"

Note that this change is recent, it was introduced in c4a165bce35edf661abaaa68c3cfd427664e5e25, so the affected versions are:

v2020.4.1b2
v2020.5.28
v2020.6.2

@techalchemy, you made this change; the repeated directory copying is causing pipenv to time out regularly on install, lock or update commands, regularly, unless we minimize what's in the project directory, and even then it just takes way too much time.

Here is a timing example with two separate pipenv versions, one before the change, one after, local project with cached files (note the size of the .git repository!). The attempt with 2020.6.2 failed, it timed out after 17 minutes:
$ du -hs . .git
3.2M	.
2.7M	.git
$ ~/.local/bin/pipenv --version
pipenv, version 2018.11.26
$ pipenv --versions
pipenv, version 2020.6.2
$ time ~/.local/bin/pipenv lock
Locking [dev-packages] dependencies…
✔ Success!
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (e0aade)!

real	0m49.705s
user	0m43.434s
sys	0m4.617s
$ time pipenv lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Locking [dev-packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/cli/command.py", line 368, in lock
    do_lock(
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/core.py", line 1103, in do_lock
    venv_resolve_deps(
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 1323, in venv_resolve_deps
    c = resolve(cmd, sp)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 1136, in resolve
    result = c.expect(u"\n", timeout=environments.PIPENV_INSTALL_TIMEOUT)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/delegator.py", line 215, in expect
    self.subprocess.expect(pattern=pattern, timeout=timeout)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 343, in expect
    return self.expect_list(compiled_pattern_list,
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 372, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/pexpect/expect.py", line 181, in expect_loop
    return self.timeout(e)
  File "/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv/vendor/pexpect/expect.py", line 144, in timeout
    raise exc
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x105e5f850>
searcher: searcher_re:
    0: re.compile('\n')
<pexpect.popen_spawn.PopenSpawn object at 0x105e5f850>
searcher: searcher_re:
    0: re.compile('\n')

real	17m22.404s
user	4m25.723s
sys	0m41.463s

Even though it timed out, it tells us a lot:

instead of 50 seconds, I had to wait over 17 minutes before there was an error.
Nearly 13 minutes  of this time was spent waiting while I/O buffers were filled and flushed while copying.

A separate Python script tracked how many directories were created:
import glob, tempfile, time
maxcount = 0
pattern = f"{tempfile.gettempdir()}/reqlib-src*"
while True:
    count = len(glob.glob(pattern))
    if count > maxcount:
        maxcount = count
        print("Current maximum directory count:", maxcount)
    time.sleep(1)

which reached:
Current maximum directory count: 85

before the pipenv lock process timed out.

I didn't dive into why, but I have noticed that it takes an extremely long time to do an update and that this has gotten worse in recent versions.

Our team is affected by this. We are either rolling back to pipenv 2018 or migrating tot poetry. The minimal activity on this thread does not give me much confidence.


[Settings] Disabled shortcuts looks the same as when enabled

PT Version: 0.1.0
Module: Settings
When the module is disabled the color of shortcuts didn't change:

cc: @niels9001

@niels9001
Can you plz update the tags.


Ek/5978 add status command

Added status command for getting information about application loading, for more details see: #5978
Command output now looks like this:
genestack/qiime-report   failed

Application jar was not loaded due to the following errors:
genestack/qiime-report JS error: this language feature is only supported for ECMASCRIPT6 mode or better: class. Use --language_in=ECMASCRIPT6 or ECMASCRIPT6_STRICT or higher to enable ES6 features.
Script loading error: unable to find script dose_response.r
genestack/qiime-report JS error: this language feature is only supported for ECMASCRIPT6 mode or better: member declaration. Use --language_in=ECMASCRIPT6 or ECMASCRIPT6_STRICT or higher to enable ES6 features.

genestack/report-viewer   loaded

Also changed interface of install command.
Note: don't merge this branch until main repo branch will be finished (consult me on any questions).
Pull request in main repo: https://github.com/genestack/genestack/pull/3269

About Use --language_in=ECMASCRIPT6: i looked to closure compiler sources and it seems that there is no way to change error messages. It uses ErrorReporter and there are several implementations, but only one can report meaningful error messages.

Please bump version to '0.15.0a2'


Adds ability to use default hasher in dev_mode for explicit key binding

Fixes #14118
Recently, we added the ability to specify hashers as _ in dev_mode, which are replaced by Blake2_128Concat. This PR allows using a default hasher in dev_mode in case of explicit key binding.
With these changes, a syntax like the below works in dev_mode but throws an error otherwise.
#[pallet::storage]
pub type Baz<T: Config> = StorageMap<Key = T::AccountId, Value = T::Balance>;

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2848260

bot fmt

bot merge


Option to select the TTS language separately ?

Would it be possible to add an option to have the timelines in one language and the TTS in another ?

What's the use case here? I guess I figured that most folks who are using one language what everything the same.

Well, as I try to read guides and run the raid sometimes with pugs, I would like to have the timeline in english to know exactly what ability they are making the call for etc...
And I would also like to the the tts in my native language as I will react faster to it.
Maybe make an option Options.TTSLanguage which is equal to Options.Language by default ?


Remove luarocks bin path

This was something I missed when I copied the asdf-lua source into the asdf-php project.

Tests fails because I'm adjusting the dependencies, but the PR is ok. Merging!


Eval simple value not work with Chicken

For example,
(+ 1 2)
42

when I put point at the ending of the first line and type C-x C-e (geiser-eval-last-sexp), it works fine by showing => 3 in the Echo area, however, when I do the same thing on the second line, it doesn't work and displays the following in *Geiser dbg* buffer:

42

Error: (car) bad argument type: 42

	Call history:

	<syntax>	  (geiser-eval (quote #f) (quote 42))
	<syntax>	  (quote #f)
	<syntax>	  (##core#quote #f)
	<syntax>	  (quote 42)
	<syntax>	  (##core#quote 42)
	<eval>	  (geiser-eval (quote #f) (quote 42))
	<eval>	  [geiser-eval] (form-has-safe-geiser? form)
	<eval>	  [form-has-safe-geiser?] (any (cut eq? (car form) <>) (quote (geiser-no-values geiser-newline geiser-completions geiser-autod...
	<eval>	  [form-has-safe-geiser?] ((##core#begin eq?) (car form) g447448)
	<eval>	  [form-has-safe-geiser?] (car form)	<--
#;98>

Weird, works for me.
What version of Chicken are you running?

~ $ csi -version

CHICKEN
(c) 2008-2017, The CHICKEN Team
(c) 2000-2007, Felix L. Winkelmann
Version 4.12.0 (rev 6ea24b6)
macosx-unix-clang-x86-64 [ 64bit manyargs dload ptables ]
compiled 2017-02-19 on yves.more-magic.net (Linux)

we are moving the project to gitlab. so please, follow-up there as needed. sorry for the inconvenience!
http://gitlab.com/jaor/geiser/issues/226


⚠ Bofrost.fr has degraded performance

In 812c709, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1242 ms

Resolved: Bofrost.fr performance has improved in 77013f2 after 6 minutes.


Documentation and model of SNAP8-ER

List of changes:

Creation of folder "microreactors" where both mrad and snap8-er model are moved
Creation of documentation for snap8-er
Addition of Serpent, griffin, and Bison input files for snap8-er core.

@sallustius

@GiudGiud, I'll work on this at the beginning of next week. Thanks for the feedback

Job VTB Documentation on 8fcf118 wanted to post the following:
View the site here
This comment will be updated on new commits.


Support more data structures

Support data structures listed on this page
https://en.wikipedia.org/wiki/List_of_data_structures

Support enum:
(#ENUM# a b)


Catch-All URL

is it possible to create catch-all url similar to flask with sanic?
from flask import Flask
app = Flask(name)
@app.route('/', defaults={'path': ''})
@app.route('/path:path')
def catch_all(path):
return 'You want path: %s' % path
if name == 'main':
app.run()

I was able to create catch-all url with sanic
from sanic import Sanic, request
from sanic.response import text

app = Sanic()

@app.route('/')
@app.route('/<path:path>')
async def catch_all(request, path=''):
    return text('You want path: %s' % path)

if __name__ == '__main__':
    app.run()


mutual learning

If we use the mutual learning network like AlignedReID, will the effect be better?

@weiyuxingchen Mutual learning is able to slightly boost the performance.


[JN-946] enrollee invitation proof-of-concept

DESCRIPTION
This enables invitation emails to be sent to participants who have data in Juniper (such as from a migration), but do not have b2c accounts, allowing them to create their linked b2c account

To enable easy testing of this feature, I needed a way to repeatedly create an enrollee who did not have a b2c account.  So I introduced the concept of "keyed" populate users.  This lets us be sure that every time we populate demo, the HDINVI enrollee will be created with a new username and so will be able to be invited without conflicting with an existing b2c account.  The email generated looks like [your<EMAIL_ADDRESS>FUTURE WORK:

b2c configuration to prefill the email address field (and maybe make it read-only)
not hardcoding all "AD_HOC" emails to prompt for subject/body text
some sort of indication in the data model (other than lastLogin) that a participantUser is (a) from a migration and (b) does not yet have a b2c account

TO TEST:  (simple manual steps for confirming core behavior -- used for pre-release checks)

redeploy ApiAdminApp
repopulate demo
make sure the ui-participant is running with b2c auth on
go to sandbox participant list https://localhost:3000/demo/studies/heartdemo/env/sandbox/participants
check-select the HDINVI enrollee, and press "Send email"
In the "email template" dropdown, select "Invitation email"
ignore the text fields, and press "send to 1 participant"
Confirm you get an email that looks like the screenshot above
click the link, confirm you are taken (after a redirect) to the "Verify your email" screen
(you won't be able to actually verify the email since demo b2c isn't yet configured)

hmmm... @connorlbark  where are you seeing there's no portal participant user?  when I do the populate demo, a portal participant user gets created for the 'invited' participant.  I wonder if there's some condition that can cause that populate to be flaky?

security hotspot is using random number generator in populate file.  psuedo random is fine there since it's dev-only


6.1.3 CAS External Integration Corporate Accounting System

By February 2025, SIMS will need to display a student’s T4A to them somehow. The T4A will reflect the sum total of all provincial funding a student has received throughout the previous calendar year.
In order to do this, SIMS has to somehow integrate with the Corporate Accounting Software (CAS), the software accounting tool which keeps a comprehensive log of all provincial funding which have been issued to a student throughout the year.
CAS typically knows about student disbursements in two ways; either it was involved directly in disbursing the money (e.g. our provincially-disbursed grants for Disabilities Services and Equipment), or there was some manual data entry done within CAS regarding disbursements which were made through other systems (e.g. letting CAS know about BC Access Grant amounts disbursed via the Feds).
Student profiles must be kept up to date in CAS in order to successfully identify students between CAS and SIMS/SFAS – this is done through some personal identifiers, as well as a unique identifier called CAS ‘supplier number’. Our SABC systems are responsible for creating processes to update CAS anytime there is a new student needing a new Supplier number, or an existing student has changed their personal information (e.g. Name or Address) to keep CAS up to date.
In summary, SIMS will need to do three things:

“Supplier Maintenance”: Ensuring each student affiliated with SABC has a supplier number which is aligned between SABC systems and CAS systems, and that CAS is made aware of relevant personal details about the student.
Account Payable Entry: Ensuring any Provincial funding which is disbursed by NSLSC is inputted into the CAS system for each applicable student ahead of December 31, 2024.
T4A Distribution: Ensuring SIMS or SABC can efficiently relay T4As which are produced by corporate finance to each individual student, by February 2025.

Supplier maintenance and Account payable entry can both be done via real-time web service calls, and the integration between SIMS and CAS must be fully implemented before December, 2024.
T4A distribution needs to be implemented by February of 2025, and the exact process is to be determined. Previous methods involving SABC legacy systems have involved shuttling pre-made PDF’s to the dashboards of each respective student.
Webservice Availability
CAS_Availability_2024.pdf
Month End: WS is Read-only from 9am on last calendar day of the month until EBS is open on the first day of the month

Day of the Week
Read-Write
Read-Only
Comment

SUN
9am
6am
Sundays 6am - 9am is a scheduled change window, so any webservice will be Read only until they finish that window, after that it turns back to read-write.  Occasionally there are CAS Sunday Change windows from 9am, so read only can extend up to noon.

MON
-
9am
WS becomes read-write somewhere between 12:00am and 2:00am the next day, depending on volumes

TUE
12am to 2am (following CGI Complete)
9pm

WED
12am to 2am (following CGI Complete)
9pm

THU
12am to 2am (following CGI Complete)
9pm

FRI
12am to 2am (following CGI Complete)
9pm
CGI run on Fridays end up until early morning of Saturday. Once done it will be read-write again, unless there is a change window that day.

SAT
12am to 2am (following CGI Complete)

9am EBS is offline (WS is still RW)

Overview of Epic ACs

[x] Set up technical connection to the CAS API (#3267)
[x] Store supplier information for students and query CAS (#3255)
[ ] Display supplier information about students to ministry (#3257)
[ ] Create new supplier information in CAS
[ ] Update SIMS supplier information from CAS
[ ] Handle the API availability and schedule/queue calls
[ ] Error handling and retry options for supplier/address
[ ] Generate AP records for Finance approval based on disbursements
[ ] Send AP records to CAS
[ ] UI for AP Activities
[ ] Error handling and retry options for AP records
[ ] Reports

Testing

[ ] Functional/UI
[ ] Pre-Live Supplier/Site Read & Write? **to confirm
[ ] Pre-Live AP Invoicing? **to confirm

Name Changes & SIN Changes


Question: Would we support client-side NIO on iOS ? If so, we should adopt Cocoapods and Carthage

Competitors of iOS are focusing on async and concurrency on low number of threads. So, what about adopting Cocoapods (or at least, make a cocoapod that has only the client parts of SwiftNIO).
Also this could facilitate making IoT apps on iOS, and would augment the chances of iOS to overcome its competitors.
But this may also not mean only making cocoapod, but also we need solid ARM support

You are mixing more than one topic here. It is already announced that NIO will gain Network.framework support, no?
Or is this just about adding Cocoapods support?
How would any of this „facilitate making IoT apps on iOS“?

Correct, SwiftNIO’s Network.framework support will be available shortly.

Our Network.framework support is now available.

The Swift NIO cocoa pods mentioned in the question, is that available anywhere or in the works?
For my own case I've made a version of Bolt-swift that uses NIO + SSL instead of BlueSock as client. With BlueSock I have CocoaPods support, and I'd love to keep CocoaPods support yet using NIO

We have not implemented any such support at this time. I think we'd merge a cocoapods spec though: we definitely would for NIOTransportServices.


Should allowed MIME / media types really be restricted to the IANA list?

The METS schema documentation says this about the MIMETYPE attribute:

The IANA MIME media type for the associated file or wrapped content. Some values for this attribute can be found on the IANA website.

Here it is specifically mentioned that some values can be found on the IANA website (because the media type has been formally registered), and the attribute also set as optional. I don't understand why this has been made more restrictive in the CSIP specification?
In a concrete case I received a package which contained an .iso file and with a MIME type of application/x-iso9660-image. This has a subtype that's part of the unregistered tree and as such not listed on the IANA website (Tika knows about the type as does shared-mime-info which lists it as an alias of application/x-cd-image, also not part of the IANA list).
I don't see a reason to restrict the types of files that can be transported in an information package to only those that have a registered media type and thus propose to relax the requirement to be the same as in METS, i.e. optional and only with a reference to the IANA list.

So I'm torn on this one as I agree with the above but believe that there is value in having an "authority" of some kind for allowed values it becomes a "free for all" where interoperability is desired. There are probably a few compromises/solutions. Two spring to mind:

Split the requirement into two, a straw man below:

Demote the current requirement to a "SHOULD", which gives a warning.
Add a new MUST requirement that insists on a well-formed MIME value only.

Add support for multiple and alternative authority lists, which is more ambitious.

While the first approach is simple and flexible, the second one is right, IMO. A single authority table doesn't make much sense when you look at different requirements, which are MIME for:

CSIP26, descriptive metadata.
CSIP40, digital provenance metadata.
CSIP53, rights metadata.
CSIP68, content files.

The restriction to IANA types for the different types of metadata files isn't that helpful. Discovering that your descriptive metadata is in an MDL file is unlikely to help much. CSIP 26,40,53 would really benefit from a stricter approach that limited the choice of metadata types. More flexible requirement conditions might still be needed, e.g. MUST have a well-formed MIME value, SHOULD be in an approved list of rights MD MIMES. CSIP68 is for content files; splitting it into two is probably the best way to start. That's got a good ramble out of my head.

Your first suggestion to split the requirement into two sounds like a good way forward to me, ideally paired with support for a more extensive list of MIME types (like from shared-mime-info and/or Tika - not sure about the relation between the two right now - as mentioned previously), as getting validation warnings for unregistered MIME types that are otherwise known doesn't sound too actionable to me.
Making a well-formed MIME string mandatory when the attribute is present also makes sense to me as METS doesn't specify that aspect explicitly (or explicitly enough? One could probably take that as implicit.).
On the point about interoperability I don't think this is really in scope in this particular case. Section 5.1 on general requirements for metadata says

Descriptive metadata and most detailed technical metadata is outside the scope of the CSIP. As such, the CSIP does not provide detailed semantic interoperability between different systems. However, as noted in Section 1.2, implementers are welcome to use the Content Information Type Specifications to achieve interoperability at a more detailed level.

Paired with 3.4.1. which states that metadata MUST conform to a standard - though a strong SHOULD would seem to be more fitting there as well to match the statement "the use of established and widely used metadata standards is highly recommended" later in that section, as well as this not really being validatable - that strikes me as sufficient for the purposes of the CSIP.
I too see value in having an authority/registry of / containing defined MIME types, restricting allowed values to known types or specific subsets thereof is then as referenced in the quote above the purpose of Content Information Types. Section 1.2 opens with:

As an interoperability standard, it must be possible to use the CSIP regardless of the type and format of the content users need to handle. At the same time, each individual content type and file format can have specific characteristics which need to be taken into account for purposes of validation, preservation and curation.

which I find very reasonable and basically what I wanted to get at in my initial description.
I'm actually also starting to wonder now why the @sip:FILEFORMAT* family of attributes is only defined for the SIP (and there only for content files) and if it makes sense to seemingly prefer MIME type info over that? Sorry if I'm opening up too many additional questions here as they come to mind.

We need to go back to the METS standard here. In the Primer ( https://www.loc.gov/standards/mets/METSPrimer.pdf ) on page 34 the definition states: "MIMETYPE (string/O): The IANA MIME media type for the associated file. Some values for this attribute can be found on the IANA website."

I'm not sure what you're saying here, @karinbredenberg? The part you quoted was also what I opened my initial post with.

Sorry, I see that I pressed comment to early so the rest of the reply got lost.
We have pointed to the IANA list that is available due to decisions made early on in the creation of CSIP. To change that requirement which is the suggestion given it will be brought to the DILCIS Board. I'm leaning towards keeping the requirements as is but change the wording so its the recommended list to use unless something else have been agreed upon.

OK, thanks for putting this on the agenda!
Opening the allowed values up by making the IANA list only the default recommendation and allowing local overriding agreements would already help. I'd still be worried about validators potentially producing warnings for MIME types not on the comparatively limited IANA list despite the possibility of additional agreements and the existence of larger (well-maintained and known) lists that could at least be referred to officially as well in some way.
As further input to the discussion and to expand on why I think this should also be made optional: I do believe that there can always be situations where data needing to be archived doesn't have a registered/known MIME type on any list (or isn't readily identified by existing tooling!) and marking packages as invalid or holding ingest back while waiting for tool improvements or new MIME type strings to be thought of (and approved) doesn't need to happen in the general case that CSIP is concerned with when it means to support inclusion "regardless of the type and format of the content users need to handle". Of course that would still often be rightfully treated as a preservation risk, but that should be able to be managed over the further lifecycle.

The issue have been discussed on todays DILCIS Board meeting and we agreed upon making it a recommendation to use the IANA list. An update of the wording will be made.

The suggestion is:

Update of wording with this PR suggestion #715

Board members acknowledgment of the issue:
Tick the box in front of you name to indicate that you have looked at the suggestion.

[ ] Karin Bredenberg (Kommunalförbundet Sydarkivera, chair)
[ ] Anders Bo Nielsen (National Archives of Denmark)
[ ] Anja Paulič (National Archives of Slovenia)
[ ] Arne-Kristian Groven (National Archives of Norway)
[ ] Gregor Zavrsnik (Geoarh)
[ ] Janet Anderson (Highbury Research & Development Ltd.)
[ ] Maya Bangerter (Swiss Federal Archives)
[ ] Miguel Ferreira (KEEP Solutions)
[ ] Stephen Mackey (Penwern Limited)
[ ] Sven Schlarb (Austrian Institute of Technology)

Voting
(Decision making will be carried out on the basis of majority voting by all eligible members of the Board. In the case of a tied vote, decisions will be made at the discretion of the Chair)
Tick the box in front of you name to say yes to the suggestion.

[ ] Karin Bredenberg (Kommunalförbundet Sydarkivera, chair)
[ ] Anders Bo Nielsen (National Archives of Denmark)
[ ] Anja Paulič (National Archives of Slovenia)
[ ] Arne-Kristian Groven (National Archives of Norway)
[ ] Gregor Zavrsnik (Geoarh)
[ ] Janet Anderson (Highbury Research & Development Ltd.)
[ ] Maya Bangerter (Swiss Federal Archives)
[ ] Miguel Ferreira (KEEP Solutions)
[ ] Stephen Mackey (Penwern Limited)
[ ] Sven Schlarb (Austrian Institute of Technology)

7 DILCIS Board members have acknowledge the issue
7 DILCIS Board members agree with the solution
The PR will be part of the next release of the specifications.

Change implemented.


Create and name a subclass of RandomVariable to represent differenced timeseries

Motivation
Classes like FirstDifferenceARProcess below could be created by composing a representation of the differencing with a representation of the underlying timeseries of $n^\mathrm{th}$ differences.
https://github.com/CDCgov/multisignal-epi-inference/blob/99cdf4272358d80966ce92007d896245d53d59e2/model/src/pyrenew/process/firstdifferencear.py#L12-L17
Steps

[ ] Choose a name for the class.
[ ] Implement it.
[ ] Reimplement the existing FirstDifferenceARProcess using it

Going to wait for #215 as it bears upon this.

Closed in https://github.com/CDCgov/PyRenew/pull/380


*: upgrade tidb version

What problem does this PR solve?

Issue Number: ref https://github.com/pingcap/tiflow/issues/4287
The TiDB repository has been updated with this PR, and the directory of the code has changed. If a third-party repository wants to import the latest TiDB code and also needs to import the tiflow code, the TiDB dependency in tiflow needs to be upgraded as well.
What is changed and how it works?
update dependency
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
`None`

/merge


Determine if we can remove the StructToStruct calls in unstructured_util.go

Describe the bug
By calling StructToStruct from GetResource() and GetLiveResource() functions, the Generation field is changing from an integer field to a float64 field. Would like to remove StructToStruct if it doesn't cause harm. Need to figure out if we can instead update tests.

Message from the maintainers:
Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

Observation: Before and after conversion of unstruc.Object via util.StructToStruct() method, here rpu type is changed from int64 to float64
Before:

After:

Are all numbers float64 after? Maybe json unmarshal just does that?

Are all numbers float64 after? Maybe json unmarshal just does that?

Yes, Json unmarshal does that, similar conversation going here, which is closed saying JSON does only know floats

This is where problem start, here it converts the pipelineRollout.Spec.Pipeline.Spec of type runtime.RawExtension to pipelineSpec of type map[string]interface{} and treat numeric value as float64, so everywhere we have to convert int64 value to float64 otherwise reflect.DeepEqual() fail the comparison due to type difference.
So there is two possible fix:

Fix the Json parser to convert number to int64 instead of float64
Fix reflect.DeepEqual() method to only compare value not type in case of number

Which call to reflect.DeepEqual() are you referring to? In a test do you mean?

Fix the Json parser to convert number to int64 instead of float64

In some cases it should be float and in some cases it should be int. It really can't know unfortunately. It makes sense that the json unmarshal just assumes float in that case.

Here's the fix I have raised, Unit test cases is passing will check the E2E test in pipeline, hopefully it should also pass.

I don't think we should do it that way unless I'm mistaken in some way. I added a comment for why.
Let me think about this.

okay, As it's an open question about JsonParser that it treat all numeric value to float64, and maybe people have followed above approach to fix it, but it depend on use-cases. OR we can think about 2nd approach also.


Archiving a GitLab project should not create a new Lagoon project

Describe the bug
At present, when you have a GitLab system hook setup for Repository update events, this fires for when a project in GitLab is archived (by design). Lagoon at present does not seem to be able to discern this archive action, from any other project action.
We are seeing issues now where Lagoon projects are deleted, then at a later point the GitLab project is archived, and then the Lagoon project is brought back to life. This is less than ideal.
An example payload from a system hook for project_update:
{
  "event_name": "project_update",
  "created_at": "2019-02-12T19:24:41+11:00",
  "updated_at": "2022-02-11T08:43:16+11:00",
  "name": "distro-testing",
  "path": "distro-testing",
  "path_with_namespace": "testing/distro-testing",
  "project_id": 345,
  "owner_name": "Testing",
  "owner_email": "",
  "project_visibility": "private"
}

So it looks like you may need to query the GitLab API to find out the current project status.
GET /api/v4/projects/345
...
  "archived": true,
...

There is an existing GitLab issue to add more information to the system hook.
To Reproduce
Steps to reproduce the behavior:

Delete the Lagoon project
Archive the corresponding GitLab repo
Notice how the Lagoon project has been re-created

Expected behavior
When a GitLab project is archived, this should not create a new project in Lagoon.

Do we go one step further and say when a project is archived in gitlab, lagoon automatically deletes it?

I think we can save that for when someone actually deletes the project in GitLab.


Spec Improvements

Increase readability of specs by moving tests to the file under files: they are associated with

Both listing tests with the file and under a separate tests section can be done in the same spec, even for the same file

supporting tag added

Same functionality as inputs, added to make specs clearer
Both supporting and inputs can be used in the same spec

Supporting files can be specified with a dict, in addition to a string or list as before:

supporting:
  - file1.txt                   # copy file to homework directory
  - [ file2.txt, file2a.txt ]   # copy file to homework directory with rename/relocation
  - file: file3.txt             # same as above but more verbose
    destination: file3a.txt
  - file: file4.txt             # same as above but using dest instead of destination
    dest: file4a.txt

Spec versions can now be combined, no need to differentiate with spec_version: 3 anymore.
Remove an extraneous guarded return that doesn't do anything anymore.

Random drive by comment
I'd like to pick one of destination or dest and only support one


question about test

Could you share more detail on how to successfully submit to the server?

I always get error like
Missing prediction for ground truth file GOPR0364_frame_000001_gt_labelTrainIds.png

Could you share more detail on how to successfully submit to the server?

Hi,
Sorry for my late response. I can show you my submission folder.
First, I prepared three folders.

The one in the middle stores all the predictions, and their filenames are like this. I thought there might be something wrong with your filename.

Since I did not explore the computation of UIoU, I just randomly put some figures in the confidence folder. The number and names of the figures are the same as that of the second folder. And the third folder contains exactly the same things as the second one.
And then directly compress the three folders. Let me know if there is anything unclear.

clear. Thanks


Issue listing Operator Hub services despite an application existing

/kind bug

What versions of software are you using?
Operating System: Linux
Output of odo version: Master (using latest: https://github.com/openshift/odo/pull/3069)
How did you run odo exactly?
Git clone springboot-ex: git clone https://github.com/odo-devfiles/springboot-ex then run the following commands:
~/openshift/springboot-ex  master ✗                                                                                118d ◒
▶ odo create java-spring-boot
Experimental mode is enabled, use at your own risk

Validation
 ✓  Checking devfile compatibility [48463ns]
 ✓  Creating a devfile component from registry: DefaultDevfileRegistry [48936ns]
 ✓  Validating devfile component [132258ns]

Please use `odo push` command to create the component with source deployed

~/openshift/springboot-ex  master ✗                                                                                118d ◒
▶ odo push

Validation
 ✓  Validating the devfile [44287ns]

Initialization
 ✓  Initializing the component [6s]

Creating Docker resources for component java-spring-boot
 ✓  Pulling image maysunfaisal/springbootbuild [561ms]
 ✓  Starting container for maysunfaisal/springbootbuild [608ms]
 ✓  Pulling image maysunfaisal/springbootruntime [594ms]
 ✓  Starting container for maysunfaisal/springbootruntime [468ms]

Syncing to component java-spring-boot
 ✓  Checking files for pushing [1ms]
 ✓  Syncing files to the component [344ms]

Executing devfile commands for component java-spring-boot
 ✓  Executing devbuild command "/artifacts/bin/build-container-full.sh" [39s]
 ✓  Executing devrun command "/artifacts/bin/start-server.sh" [1s]

Pushing devfile component java-spring-boot
 ✓  Changes successfully pushed to component

~/openshift/springboot-ex  master ✗                                                                                118d ⚑ ◒  ⍉
▶ odo service list
 ✗  Please specify the application name and project name
Or use the command from inside a directory containing an odo component.

~/openshift/springboot-ex  master ✗                                                                                118d ⚑ ◒
▶ odo service create --from-file foobar.yaml
Deploying service of type: EtcdCluster
 ✓  Deploying service [7ms]
 ✓  Service 'foobar2' was created

Progress of the provisioning will not be reported and might take a long time
You can see the current status by executing 'odo service list'
Optionally, link  to your component by running: 'odo link <component-name>'

~/openshift/springboot-ex  master ✗                                                                                118d ⚑ ◒  ⍉
▶ odo service list
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x159082e]

goroutine 1 [running]:
github.com/openshift/odo/pkg/kclient.(*Client).GetClusterServiceVersionList(0x0, 0x18da900, 0x190e0e0, 0xc000745640)
        /home/wikus/syncthing/files/dev/go/src/github.com/openshift/odo/pkg/kclient/operators.go:18 +0x4e
github.com/openshift/odo/pkg/service.ListOperatorServices(0x0, 0xc0002ea868, 0x195fbe0, 0xc0002ea840, 0x1f0a180, 0xc000795658)
        /home/wikus/syncthing/files/dev/go/src/github.com/openshift/odo/pkg/service/service.go:230 +0x83
github.com/openshift/odo/pkg/odo/cli/service.(*ServiceListOptions).Run(0xc0004c93c0, 0x0, 0x0)
        /home/wikus/syncthing/files/dev/go/src/github.com/openshift/odo/pkg/odo/cli/service/list.go:72 +0x5b
github.com/openshift/odo/pkg/odo/genericclioptions.GenericRun(0x1f64420, 0xc0004c93c0, 0xc000417900, 0x2ff9850, 0x0, 0x0)
        /home/wikus/syncthing/files/dev/go/src/github.com/openshift/odo/pkg/odo/genericclioptions/runnable.go:31 +0x13c
github.com/openshift/odo/pkg/odo/cli/service.NewCmdServiceList.func1(0xc000417900, 0x2ff9850, 0x0, 0x0)
        /home/wikus/syncthing/files/dev/go/src/github.com/openshift/odo/pkg/odo/cli/service/list.go:129 +0x5e
github.com/openshift/odo/vendor/github.com/spf13/cobra.(*Command).execute(0xc000417900, 0x2ff9850, 0x0, 0x0, 0xc000417900, 0x2ff9850)
        /home/wikus/syncthing/files/dev/go/src/github.com/openshift/odo/vendor/github.com/spf13/cobra/command.go:830 +0x2aa
github.com/openshift/odo/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0003cdb80, 0x1d5e0d8, 0xc00004b6e0, 0x2)
        /home/wikus/syncthing/files/dev/go/src/github.com/openshift/odo/vendor/github.com/spf13/cobra/command.go:914 +0x2fb
github.com/openshift/odo/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /home/wikus/syncthing/files/dev/go/src/github.com/openshift/odo/vendor/github.com/spf13/cobra/command.go:864
main.main()
        /home/wikus/syncthing/files/dev/go/src/github.com/openshift/odo/cmd/odo/odo.go:67 +0x32a

Even with:
odo service create etcdoperator.v0.9.4-clusterwide --crd EtcdCluster

It still doesn't work ;(
Actual behavior
Unable to list any services..
Expected behavior
Being to list services.
Any logs, error output, etc?

ping @dharmit

~/openshift/springboot-ex  master ✗                                                                                118d ⚑ ◒
▶ odo service create --from-file foobar.yaml
Deploying service of type: EtcdCluster
✓  Deploying service [7ms]
✓  Service 'foobar2' was created

Can you share the foobar.yaml that was used to create a service?

Even with:
odo service create etcdoperator.v0.9.4-clusterwide --crd EtcdCluster

What doesn't work? odo service list? What error do you encounter?

~/openshift/springboot-ex  master ✗                                                                                118d ⚑ ◒
arrow_forward odo service create --from-file foobar.yaml
Deploying service of type: EtcdCluster
✓  Deploying service [7ms]
✓  Service 'foobar2' was created

Can you share the foobar.yaml that was used to create a service?

Even with:
odo service create etcdoperator.v0.9.4-clusterwide --crd EtcdCluster

What doesn't work? odo service list? What error do you encounter?

Sorry,
foobar.yaml is from:
odo service create etcdoperator.v0.9.4-clusterwide --crd EtcdCluster --dry-run > foobar.yaml

Any updates on this issue @cdrage @dharmit


Sharing process issue templates

These templates should complement the new process and policy pages in https://github.com/agrc/gis.utah.gov/pull/1627. I'm leaving this as a draft until that PR is finalized to allow for any changes that result from policy discussions.
Closes #88

Is this because it's assumed that the person creating the issue is the one to complete these tasks?

Yes, I wanted to reduce the steps required for them.

Is this because it's assumed that the person creating the issue is the one to complete these tasks?

Yes, I wanted to reduce the steps required for them.

When asked, jake decided to close this. Maybe it will come up again with the MATT efforts.

These unfinished issue templates are referenced in our current and new website. We should resolve and merge this pr.


Animate only specific section

Hi,
It will be nice if we can animate specific section only when there are multiple visible sections.

Hey @alimoon88. Thanks for the suggestion. Unfortunately the API for TableFlip works by looking at the current visible cells only, rather than specifically dealing with any IndexPaths, rows, or sections. I can think about restructuring this at another point in time if this is something you feel is quite important. Alternatively if you want to file a PR, I'd be happy to read it over and think through it.
Let me know!

Hi @mergesort, Thanks for your quick reply, just after adding the suggestion I modified the code and did exactly what I want, animating specific visible section cells.
UITableView+TableFlip.swift.zip

Hey @alimoon88. This actually looks like a pretty good solution, thanks! I'll play with it a bit and try and find some time to implement it.
I wish I could give you some fake internet points, but instead I'll just say, thanks again!

Hey @alimoon88. It's a little delayed, but I went ahead and implemented the changes you requested in this commit.
My implementation is a tiny bit different than yours, but your code was a terrific foundation, so thank you very much for the work. If you're using CocoaPods, you can also update to version 1.0.1, to get the changes in your app.
Thanks again!


Make it easier to run on the cloud

It would be great if it was easier to run large simulations on the cloud.
It would be pretty easy to set up a custom docker image with Julia installed and ideally a sysimage there too with the OpticSim dependencies in - this would be hugely valuable in itself for cloud execution.
To go even further we could then have a script within the default package which submits runs e.g. to Azure ML using this docker image and whatever command the user wants to execute.
AzureML has a nice thing where it uploads a snapshot of the source directory, so it is possible to modify the source code and run jobs with those modifications.
e.g.
OpticSim.submit_job_to_AML(,aml_workspace, relpath_to_entry_script, args_for_script)

I doubt there is a julia wrapper for AML API, but we could probably call the python API somehow...
Certainly not sure on the details here, but enabling cloud runs seems like something that should be reasonably high priority

This ties in nicely with our push towards Pluto notebooks as we won't want to do heavy (or even light) simulations on a web server. I'll look into it when I can.

@friggog were you going to take a crack at this? Sounds like you have the most knowledge about AzureML.

Yeah I’m happy to take a look, at least to get the basics running and then maybe you can add a more comprehensive framework on top of that
Sent from my phone

On 30 Mar 2021, at 19:51, BrianGun @.***> wrote:

@friggog were you going to take a crack at this? Sounds like you have the most knowledge about AzureML.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Resolved by #71


unknown bit in zynq7 harness

Using database https://github.com/mithro/prjxray-db/commit/88d9a13ebdf431ec9bd94377817daf4d46f7a938
Writing bitstream ./design.bit...
INFO: [Vivado 12-1842] Bitgen Completed Successfully.
INFO: [Project 1-118] WebTalk data collection is enabled (User setting is ON. Install Setting is ON.).
INFO: [Common 17-186] '/usr/local/google/home/tansell/github/SymbiFlow/prjxray/minitests/roi_harness/build/design/usage_statistics_webtalk.xml' has been successfully sent to Xilinx on Tue Mar 26 18:26:29 2019. For additional details about this file, please refer to the WebTalk help file at /opt/Xilinx/Vivado/2017.2/doc/webtalk_introduction.html.
9 Infos, 5 Warnings, 0 Critical Warnings and 0 Errors encountered.
write_bitstream completed successfully
write_bitstream: Time (s): cpu = 00:00:07 ; elapsed = 00:00:06 . Memory (MB): peak = 2350.082 ; gain = 332.453 ; free physical = 17416 ; free virtual = 39424
INFO: [Common 17-206] Exiting Vivado at Tue Mar 26 18:26:29 2019...
++ fgrep CRITICAL vivado.log
+ test -z ''
+ /usr/local/google/home/tansell/github/SymbiFlow/prjxray/build/tools/bitread --part_file /usr/local/google/home/tansell/github/SymbiFlow/prjxray/database/zynq7/xc7z010clg400-1.yaml -F 0x00000000:0xffffffff -o design.bits -z -y design.bit
Bitstream size: 2083839 bytes
Config size: 520922 words
Number of configuration frames: 5144
DONE
+ python3 /usr/local/google/home/tansell/github/SymbiFlow/prjxray/utils/bit2fasm.py --verbose design.bit
+ python3 /usr/local/google/home/tansell/github/SymbiFlow/prjxray/utils/fasm2frames.py design.fasm design.frm
+ PYTHONPATH=/usr/local/google/home/tansell/github/SymbiFlow/prjxray:/usr/local/google/home/tansell/github/SymbiFlow/prjxray/third_party/fasm:/usr/local/google/home/tansell/github/SymbiFlow/prjxray:/usr/local/google/home/tansell/github/SymbiFlow/prjxray/third_party/fasm:/usr/local/google/home/tansell/github/SymbiFlow/prjxray:/usr/local/google/home/tansell/github/SymbiFlow/prjxray/third_party/fasm:/usr/local/google/home/tansell/github/SymbiFlow/prjxray:/usr/local/google/home/tansell/github/SymbiFlow/prjxray/third_party/fasm:/usr/local/buildtools/current/sitecustomize:/usr/local/google/home/tansell/github/SymbiFlow/prjxray/utils
+ python3 ../create_design_json.py --design_info_txt design_info.txt --design_txt design.txt --pad_wires design_pad_wires.txt --design_fasm design.fasm
Traceback (most recent call last):
  File "../create_design_json.py", line 114, in <module>
    main()
  File "../create_design_json.py", line 92, in main
    unknown_base_address)
AssertionError: Found unknown bit in base address 0x00001380
Makefile:14: recipe for target 'build/design.bit' failed
make: *** [build/design.bit] Error 1

@mithro Did you update the zync databases after https://github.com/SymbiFlow/prjxray/pull/727 was merged?

{ unknown_bit = "0000139a_0_8", unknown_segment = "0x00001380", unknown_segbit = "26_08" }
{ unknown_bit = "0000139a_42_25", unknown_segment = "0x00001380", unknown_segbit = "26_1369" }
{ unknown_bit = "0000139a_42_28", unknown_segment = "0x00001380", unknown_segbit = "26_1372" }
{ unknown_bit = "0000139b_24_12", unknown_segment = "0x00001380", unknown_segbit = "27_780" }
{ unknown_bit = "0000139b_0_14", unknown_segment = "0x00001380", unknown_segbit = "27_14" }
{ unknown_bit = "00001b1b_49_25", unknown_segment = "0x00001b00", unknown_segbit = "27_1593" }
{ unknown_bit = "00001b1b_49_23", unknown_segment = "0x00001b00", unknown_segbit = "27_1591" }
{ unknown_bit = "00001b1b_50_25", unknown_segment = "0x00001b00", unknown_segbit = "27_1625" }
{ unknown_bit = "00001b1b_49_29", unknown_segment = "0x00001b00", unknown_segbit = "27_1597" }
{ unknown_bit = "00001b1b_49_31", unknown_segment = "0x00001b00", unknown_segbit = "27_1599" }
{ unknown_bit = "00001ba6_0_22", unknown_segment = "0x00001b80", unknown_segbit = "38_22" }
{ unknown_bit = "00001ba6_1_22", unknown_segment = "0x00001b80", unknown_segbit = "38_54" }
{ unknown_bit = "00001ba6_53_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1718" }
{ unknown_bit = "00001ba6_61_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1974" }
{ unknown_bit = "00001ba6_54_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1750" }
{ unknown_bit = "00001ba6_62_22", unknown_segment = "0x00001b80", unknown_segbit = "38_2006" }
{ unknown_bit = "00001ba7_53_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1719" }
{ unknown_bit = "00001ba7_61_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1975" }
{ unknown_bit = "00001ba7_54_17", unknown_segment = "0x00001b80", unknown_segbit = "39_1745" }
{ unknown_bit = "00001ba7_62_17", unknown_segment = "0x00001b80", unknown_segbit = "39_2001" }
{ unknown_bit = "00001ba7_0_21", unknown_segment = "0x00001b80", unknown_segbit = "39_21" }
{ unknown_bit = "00001ba7_0_23", unknown_segment = "0x00001b80", unknown_segbit = "39_23" }
{ unknown_bit = "00001ba7_54_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1751" }
{ unknown_bit = "00001ba7_62_23", unknown_segment = "0x00001b80", unknown_segbit = "39_2007" }
{ unknown_bit = "00001ba7_54_29", unknown_segment = "0x00001b80", unknown_segbit = "39_1757" }
{ unknown_bit = "00001ba7_62_29", unknown_segment = "0x00001b80", unknown_segbit = "39_2013" }
{ unknown_bit = "00001ba7_1_17", unknown_segment = "0x00001b80", unknown_segbit = "39_49" }
{ unknown_bit = "00001ba7_53_21", unknown_segment = "0x00001b80", unknown_segbit = "39_1717" }
{ unknown_bit = "00001ba7_61_21", unknown_segment = "0x00001b80", unknown_segbit = "39_1973" }
{ unknown_bit = "00001ba7_1_23", unknown_segment = "0x00001b80", unknown_segbit = "39_55" }
{ unknown_bit = "00001ba7_1_29", unknown_segment = "0x00001b80", unknown_segbit = "39_61" }
{ unknown_bit = "0040139d_73_0", unknown_segment = "0x00401380", unknown_segbit = "29_2336" }
{ unknown_bit = "00401ba8_89_30", unknown_segment = "0x00401b80", unknown_segbit = "40_2878" }
{ unknown_bit = "00401ba8_90_22", unknown_segment = "0x00401b80", unknown_segbit = "40_2902" }
{ unknown_bit = "00401ba8_89_22", unknown_segment = "0x00401b80", unknown_segbit = "40_2870" }
{ unknown_bit = "00401ba9_90_29", unknown_segment = "0x00401b80", unknown_segbit = "41_2909" }
{ unknown_bit = "00401ba9_89_23", unknown_segment = "0x00401b80", unknown_segbit = "41_2871" }
{ unknown_bit = "00401ba9_90_17", unknown_segment = "0x00401b80", unknown_segbit = "41_2897" }
{ unknown_bit = "00401ba9_89_21", unknown_segment = "0x00401b80", unknown_segbit = "41_2869" }
{ unknown_bit = "00401ba9_90_23", unknown_segment = "0x00401b80", unknown_segbit = "41_2903" }
{ unknown_bit = "00401ba9_89_29", unknown_segment = "0x00401b80", unknown_segbit = "41_2877" }
{ unknown_bit = "0040111d_68_29", unknown_segment = "0x00401100", unknown_segbit = "29_2205" }

Looks like a base address problem.

@litghost Yes, I updated the database based on c7d7e9d7ade70bd47f969f9713d9a6e4defbd53a

I did a small crosscheck of the missing bits and the current tilegrid.json:

# In frame 0x0000139b 1 bits were not converted.
{ unknown_bit = "0000139b_24_12", unknown_segment = "0x00001380", unknown_segbit = "27_780" }

Missing bit: CLK_BUFG_REBUF_X82Y65 27_12

# In frame 0x0040139d 1 bits were not converted.
{ unknown_bit = "0040139d_73_0", unknown_segment = "0x00401380", unknown_segbit = "29_2336" }

Missing bit: CLK_BUFG_REBUF_X82Y38 29_0

# In frame 0x00001ba6 6 bits were not converted.
{ unknown_bit = "00001ba6_0_22", unknown_segment = "0x00001b80", unknown_segbit = "38_22" }
{ unknown_bit = "00001ba6_1_22", unknown_segment = "0x00001b80", unknown_segbit = "38_54" }

Missing bits:
RIOB33_SING_X31Y50 38_22
RIOB33_SING_X31Y50 38_54

{ unknown_bit = "00001ba6_53_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1718" }
{ unknown_bit = "00001ba6_61_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1974" }

Missing bits:
RIOB33_X31Y75 38_86
RIOB33_X31Y79 38_86

{ unknown_bit = "00001ba6_54_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1750" }
{ unknown_bit = "00001ba6_62_22", unknown_segment = "0x00001b80", unknown_segbit = "38_2006" }

Missing bits:
RIOB33_X31Y75 38_118
RIOB33_X31Y79 38_118

# In frame 0x00001ba7 15 bits were not converted.
{ unknown_bit = "00001ba7_53_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1719" }
{ unknown_bit = "00001ba7_61_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1975" }
{ unknown_bit = "00001ba7_54_17", unknown_segment = "0x00001b80", unknown_segbit = "39_1745" }
{ unknown_bit = "00001ba7_62_17", unknown_segment = "0x00001b80", unknown_segbit = "39_2001" }
{ unknown_bit = "00001ba7_0_21", unknown_segment = "0x00001b80", unknown_segbit = "39_21" }
{ unknown_bit = "00001ba7_0_23", unknown_segment = "0x00001b80", unknown_segbit = "39_23" }
{ unknown_bit = "00001ba7_54_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1751" }
{ unknown_bit = "00001ba7_62_23", unknown_segment = "0x00001b80", unknown_segbit = "39_2007" }
{ unknown_bit = "00001ba7_54_29", unknown_segment = "0x00001b80", unknown_segbit = "39_1757" }
{ unknown_bit = "00001ba7_62_29", unknown_segment = "0x00001b80", unknown_segbit = "39_2013" }
{ unknown_bit = "00001ba7_1_17", unknown_segment = "0x00001b80", unknown_segbit = "39_49" }
{ unknown_bit = "00001ba7_53_21", unknown_segment = "0x00001b80", unknown_segbit = "39_1717" }
{ unknown_bit = "00001ba7_61_21", unknown_segment = "0x00001b80", unknown_segbit = "39_1973" }
{ unknown_bit = "00001ba7_1_23", unknown_segment = "0x00001b80", unknown_segbit = "39_55" }
{ unknown_bit = "00001ba7_1_29", unknown_segment = "0x00001b80", unknown_segbit = "39_61" }

Other RIOB tile related bits are missing

# In frame 0x00401ba8 3 bits were not converted.
{ unknown_bit = "00401ba8_89_30", unknown_segment = "0x00401b80", unknown_segbit = "40_2878" }
{ unknown_bit = "00401ba8_90_22", unknown_segment = "0x00401b80", unknown_segbit = "40_2902" }
{ unknown_bit = "00401ba8_89_22", unknown_segment = "0x00401b80", unknown_segbit = "40_2870" }

RIOB tiles in the southern half

# In frame 0x0040111d 1 bits were not converted.
{ unknown_bit = "0040111d_68_29", unknown_segment = "0x00401100", unknown_segbit = "29_2205" }

Couldn't find any tile in tilegrid.json that would have more than 28 frames with this baseaddress.

# In frame 0x00401ba9 6 bits were not converted.
{ unknown_bit = "00401ba9_90_29", unknown_segment = "0x00401b80", unknown_segbit = "41_2909" }
{ unknown_bit = "00401ba9_89_23", unknown_segment = "0x00401b80", unknown_segbit = "41_2871" }
{ unknown_bit = "00401ba9_90_17", unknown_segment = "0x00401b80", unknown_segbit = "41_2897" }
{ unknown_bit = "00401ba9_89_21", unknown_segment = "0x00401b80", unknown_segbit = "41_2869" }
{ unknown_bit = "00401ba9_90_23", unknown_segment = "0x00401b80", unknown_segbit = "41_2903" }
{ unknown_bit = "00401ba9_89_29", unknown_segment = "0x00401b80", unknown_segbit = "41_2877" }

Again RIOB tiles in the southern half
Summarizing:
We are missing few bits for CLK_BUFG_REBUF, RIOB33 and RIOB33_SINGLE.
There is also one bit with minor frame address above the frame count for any known segment in tilegrid.json:

# In frame 0x0040111d 1 bits were not converted.
{ unknown_bit = "0040111d_68_29", unknown_segment = "0x00401100", unknown_segbit = "29_2205"

I guess that for the RIOBs we could modify the harness design to use the features we already have solved the bits for.

@tmichalak I would start by figuring out that bit with no frame address, as that likely needs to be solved first.  The maybe figure out what is going on in the clock column.  I'll warn you that it may look like a CLK_BUFG_REBUF, but my confidence in the address/word ranges for the clock column is not the highest.

Also note that bits in the IOB may also belong to the IOI, as we are currently have merged those sections for expedience.

IOI? Could you say a bit more about it?

The tile next to IOB, containing the ILOGIC and OLOGIC sites.

Yepp, I suspected you might have meant those. Wanted to be sure.

So for the mystical unknown bit:

# In frame 0x0040111d 1 bits were not converted.
{ unknown_bit = "0040111d_68_29", unknown_segment = "0x00401100", unknown_segbit = "29_2205"

It turns out when the BITSTREAM.GENERAL.PERFRAMECRC property is set to YES becomes:

# In frame 0x0040111b 1 bits were not converted.
{ unknown_bit = "0040111b_68_29", unknown_segment = "0x00401100", unknown_segbit = "27_2205" }

And this bit can be found in each and every single bitstream in our zynq fuzzers.
This address falls into INT_L region, but there are no bits for it because I guess it's always on.

{ unknown_bit = "00001ba6_0_22", unknown_segment = "0x00001b80", unknown_segbit = "38_22" }
{ unknown_bit = "00001ba6_1_22", unknown_segment = "0x00001b80", unknown_segbit = "38_54" }
{ unknown_bit = "00001ba6_53_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1718" }
{ unknown_bit = "00001ba6_61_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1974" }
{ unknown_bit = "00001ba6_54_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1750" }
{ unknown_bit = "00001ba6_62_22", unknown_segment = "0x00001b80", unknown_segbit = "38_2006" }
{ unknown_bit = "00001ba7_53_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1719" }
{ unknown_bit = "00001ba7_61_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1975" }
{ unknown_bit = "00001ba7_54_17", unknown_segment = "0x00001b80", unknown_segbit = "39_1745" }
{ unknown_bit = "00001ba7_62_17", unknown_segment = "0x00001b80", unknown_segbit = "39_2001" }
{ unknown_bit = "00001ba7_0_21", unknown_segment = "0x00001b80", unknown_segbit = "39_21" }
{ unknown_bit = "00001ba7_0_23", unknown_segment = "0x00001b80", unknown_segbit = "39_23" }
{ unknown_bit = "00001ba7_54_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1751" }
{ unknown_bit = "00001ba7_62_23", unknown_segment = "0x00001b80", unknown_segbit = "39_2007" }
{ unknown_bit = "00001ba7_54_29", unknown_segment = "0x00001b80", unknown_segbit = "39_1757" }
{ unknown_bit = "00001ba7_62_29", unknown_segment = "0x00001b80", unknown_segbit = "39_2013" }
{ unknown_bit = "00001ba7_1_17", unknown_segment = "0x00001b80", unknown_segbit = "39_49" }
{ unknown_bit = "00001ba7_53_21", unknown_segment = "0x00001b80", unknown_segbit = "39_1717" }
{ unknown_bit = "00001ba7_61_21", unknown_segment = "0x00001b80", unknown_segbit = "39_1973" }
{ unknown_bit = "00001ba7_1_23", unknown_segment = "0x00001b80", unknown_segbit = "39_55" }
{ unknown_bit = "00001ba7_1_29", unknown_segment = "0x00001b80", unknown_segbit = "39_61" }

These bits have been solved for artix:

RIOB33.IOB_Y0.IN_ONLY 38_118 39_119 39_125
RIOB33.IOB_Y0.LVCMOS12_LVCMOS15_LVCMOS18.IN 38_86 39_85 !39_87 39_113
RIOB33.IOB_Y0.LVCMOS25_LVCMOS33_LVTTL.IN 38_86 39_85 39_87 39_113

What remains is analyzing why they are not computed for zynq.

So all of the unknown bits are known for artix except the one related to the   BITSTREAM.GENERAL.PERFRAMECRC setting:

# In frame 0x0040111d 1 bits were not converted.
{ unknown_bit = "0040111d_68_29", unknown_segment = "0x00401100", unknown_segbit = "29_2205"

This bit seems to be Zynq specific, as I didn't notice it in Artix on Kintex.
To summarize the rest of the unknown bits I had with the latest roi_harness run for zynq7-swbut:

{ unknown_bit = "00001ba6_0_22", unknown_segment = "0x00001b80", unknown_segbit = "38_22" }
{ unknown_bit = "00001ba6_1_22", unknown_segment = "0x00001b80", unknown_segbit = "38_54" }
{ unknown_bit = "00001ba6_53_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1718" }
{ unknown_bit = "00001ba6_61_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1974" }
{ unknown_bit = "00001ba6_54_22", unknown_segment = "0x00001b80", unknown_segbit = "38_1750" }
{ unknown_bit = "00001ba6_62_22", unknown_segment = "0x00001b80", unknown_segbit = "38_2006" }
{ unknown_bit = "00001ba7_53_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1719" }
{ unknown_bit = "00001ba7_61_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1975" }
{ unknown_bit = "00001ba7_54_17", unknown_segment = "0x00001b80", unknown_segbit = "39_1745" }
{ unknown_bit = "00001ba7_62_17", unknown_segment = "0x00001b80", unknown_segbit = "39_2001" }
{ unknown_bit = "00001ba7_0_21", unknown_segment = "0x00001b80", unknown_segbit = "39_21" }
{ unknown_bit = "00001ba7_0_23", unknown_segment = "0x00001b80", unknown_segbit = "39_23" }
{ unknown_bit = "00001ba7_54_23", unknown_segment = "0x00001b80", unknown_segbit = "39_1751" }
{ unknown_bit = "00001ba7_62_23", unknown_segment = "0x00001b80", unknown_segbit = "39_2007" }
{ unknown_bit = "00001ba7_54_29", unknown_segment = "0x00001b80", unknown_segbit = "39_1757" }
{ unknown_bit = "00001ba7_62_29", unknown_segment = "0x00001b80", unknown_segbit = "39_2013" }
{ unknown_bit = "00001ba7_1_17", unknown_segment = "0x00001b80", unknown_segbit = "39_49" }
{ unknown_bit = "00001ba7_53_21", unknown_segment = "0x00001b80", unknown_segbit = "39_1717" }
{ unknown_bit = "00001ba7_61_21", unknown_segment = "0x00001b80", unknown_segbit = "39_1973" }
{ unknown_bit = "00001ba7_1_23", unknown_segment = "0x00001b80", unknown_segbit = "39_55" }
{ unknown_bit = "00001ba7_1_29", unknown_segment = "0x00001b80", unknown_segbit = "39_61" }

Solved for Artix:

RIOB33.IOB_Y0.IN_ONLY 38_118 39_119 39_125
RIOB33.IOB_Y0.LVCMOS25_LVCMOS33_LVTTL.IN 38_86 39_85 39_87 39_113

 In frame 0x0000139a 2 bits were not converted.
{ unknown_bit = "0000139a_47_12", unknown_segment = "0x00001380", unknown_segbit = "26_1516" }
{ unknown_bit = "0000139a_50_23", unknown_segment = "0x00001380", unknown_segbit = "26_1623" }
{ unknown_bit = "0000139b_47_10", unknown_segment = "0x00001380", unknown_segbit = "27_1514" }
{ unknown_bit = "0000139c_50_18", unknown_segment = "0x00001380", unknown_segbit = "28_1618" }

Solved for Artix:

CLK_HROW_TOP_R.CLK_HROW_CK_MUX_OUT_R0.CLK_HROW_R_CK_GCLK16 26_172  27_170
CLK_HROW_TOP_R.CLK_HROW_R_CK_GCLK16_ACTIVE 26_279
CLK_HROW_TOP_R.CLK_HROW_CK_IN_R0_ACTIVE 28_274

# In frame 0x0000139b 2 bits were not converted.
{ unknown_bit = "0000139b_24_12", unknown_segment = "0x00001380", unknown_segbit = "27_780" }

Solved for Artix:

CLK_BUFG_REBUF.CLK_BUFG_REBUF_R_CK_GCLK16_TOP.CLK_BUFG_REBUF_R_CK_GCLK16_BOT 27_12

I need to investigate why 030-iob, 041-clk-hrow-pips and 043-clk-rebuf-pips don't solve the bits for zynq while solving it for artix.

The unknown bits calculated with 030-iob(#820) will be solved after #823 is merged.
We are still missing few BUFG_REBUF PIPs calcualted in 043-clk-rebuf-pips.
To be precise these four PIPs are calculated for Artix, but are not for Zynq:

CLK_BUFG_REBUF.CLK_BUFG_REBUF_R_CK_GCLK0_TOP.CLK_BUFG_REBUF_R_CK_GCLK0_BOT 27_13
CLK_BUFG_REBUF.CLK_BUFG_REBUF_R_CK_GCLK16_TOP.CLK_BUFG_REBUF_R_CK_GCLK16_BOT 27_12
CLK_BUFG_REBUF.CLK_BUFG_REBUF_R_CK_GCLK19_TOP.CLK_BUFG_REBUF_R_CK_GCLK19_BOT 27_60
CLK_BUFG_REBUF.CLK_BUFG_REBUF_R_CK_GCLK21_BOT.CLK_BUFG_REBUF_R_CK_GCLK21_TOP 27_94

Once these are addressed we will have to fix  041-clk-hrow-pips to get the few missing bits and then we should be good.

All bits, except some Zynq specific which are enabled in all Zynq's fuzzer bitstreams, are known.
The always on bits are:
In frame 0x0040111b 1 bits were not converted.
{ unknown_bit = "0040111b_68_29", unknown_segment = "0x00401100", unknown_segbit = "27_2205" }
In frame 0x0040111a 2 bits were not converted.
{ unknown_bit = "0040111a_68_30", unknown_segment = "0x00401100", unknown_segbit = "26_2206" }
{ unknown_bit = "0040111a_68_31", unknown_segment = "0x00401100", unknown_segbit = "26_2207" }

What tiles do you believe these bits belong too?

Hard to tell. These bits were present in every single specimen generated for Zynq7. And one of them depended on the PERFRAMECRC setting as I described here: https://github.com/SymbiFlow/prjxray/issues/746#issuecomment-483615681

If one of the bits is correlated with PERFRAMECRC, simply write a fuzzer to document that?  There are several CMT tiles that are good candidates for fabric wide configuration, if I had to guess it lives there.

I have created a minitest (https://github.com/antmicro/prjxray/tree/bitstream_properties) which shows that the missing bits are present in an almost empty bitstream.
Almost empty, because we need at least a LUT to generate a bitstream, but constraining the LUT to various places didn't affect the presence of the three bits.
My theory is that the bits belong to bitstream properties. For example when BITSTREAM.CONFIG.USERID which is by default set to 0xFFFFFFFF is changed to 0x0 we get the following bits:

0040111a_068_00
0040111a_068_01
0040111a_068_02
0040111a_068_03
0040111a_068_04
0040111a_068_05
0040111a_068_06
0040111a_068_07
0040111a_068_08
0040111a_068_09
0040111a_068_10
0040111a_068_11
0040111a_068_12
0040111a_068_13
0040111a_068_14
0040111a_068_15
0040111a_068_30 << missing bit 1
0040111a_068_31 << missing bit 2
0040111b_068_00
0040111b_068_01
0040111b_068_02
0040111b_068_03
0040111b_068_04
0040111b_068_05
0040111b_068_06
0040111b_068_07
0040111b_068_08
0040111b_068_09
0040111b_068_10
0040111b_068_11
0040111b_068_12
0040111b_068_13
0040111b_068_14
0040111b_068_15
0040111b_068_29 << missing bit 3

The pattern here is easy to spot - LSB in USERID set to 0 results in 0040111a_068_00, LSB+1 set to 0 results in 0040111b_068_00.
Unfortunately, USERID is 32bit so we only get to bit 15 in word 68 for a given frame.
I tried out all other bitstream properties and saw bits changing in the same frame address, but not where we need it.
I compared it with Zynq 7020 to see if those bits are part related and as it turns out apart from the frame address the word and bits remained the same.
This must be something property related or part specific...

@tmichalak Change your minitest to have no inputs or outputs or clocks.  I believe that toggles 2 of the unknown bits.

@litghost: Having the pins or not doesn't influence the presence of these bits. Again, I don't think they rely on the design, but on the bitstream properties.
I reran all fuzzers for zynq trying to find a case in which these 3 bits wouldn't show up in the bitstream and guess what - none. All bits file that were generated included the 3 bits.
I updated the minitest to be able to generate a bitstream with random properties and had the same result. Worth noting though is that with the following bitstream setting: set_property BITSTREAM.GENERAL.DISABLE_JTAG YES [current_design] the bitstream had bits very close to the ones we are looking for, namely:

0040111a_068_26 << DISABLE_JTAG
0040111a_068_30 << missing
0040111a_068_31 << missing
0040111b_068_27 << DISABLE_JTAG
0040111b_068_28 << DISABLE_JTAG
0040111b_068_29 << missing

Based on these finding I decided to start creating a fuzzer, the initial commit is on the branch.
The only difference in fuzzing I see compared to all other is that here we don't have an association to any tile as these features are bitstream related.

@litghost
I think that we could assume that the missing bits are in fact always on.
I experimented by artificially injecting bitstream config bits.
In order to not break the existing flow I added an artificial tile_type called bitstream_config:
tile_type_BITSTREAM_CONFIG.json:

{
    "tile_type": "BITSTREAM_CONFIG"
}

and placed it on grid coordinates that do not exist on the fabric:

    "BITSTREAM": {
        "bits": {
            "CLB_IO_CLK": {
                "baseaddr": "0x00401100",
                "frames": 28,
                "offset": 68,
                "words": 1
            }
        },
        "grid_x": 128,
        "grid_y": 0,
        "sites": {
        },
        "type": "BITSTREAM_CONFIG"
    }

The tilegrid fuzzer will be responsible for calculating the base address and adding this to the tilegrid with the fake grid coordinates.
The next step will be to add the bitstream_config fuzzer that will inject the fake always on bits:

BITSTREAM_CONFIG.ALWAYS_ON_1 26_30
BITSTREAM_CONFIG.ALWAYS_ON_2 26_31
BITSTREAM_CONFIG.ALWAYS_ON_3 27_29

The initial version of the fuzzer is already on a branch: https://github.com/antmicro/prjxray/commits/bitstream_properties
However, it will have to be adjusted to the new fake tile type.
The bitstream produced from the fasm file with (as well as without) the bits asserted using fasm2frames and xcframes2bit works fine, at least for the counter test.
However I am not sure if the tile outside of the grid will not explode somewhere in symbiflow-arch-defs. That is something that I want to see next.

In order to not break the existing flow I added an artificial tile_type called bitstream_config:
tile_type_BITSTREAM_CONFIG.json:

Why not just use CFG_CENTER_MID over a purely synthetic tile?  As far as I can tell CFG_CENTER_MID is the same.

Like I said elsewhere, at least one of the bits is not always constant:
{ unknown_bit = "0040111b_68_29", unknown_segment = "0x00401100", unknown_segbit = "27_2205" }`

Appears to toggle.   I've attached an example verilog/dcp/bit/fasm example.
toggle_ex.tgz.zip

@litghost
Hmm...I was always looking for the bits in the bits files. And in the archive you sent I also see these three bites across all of the bits files. However it's not always the case for fasm.
How come we have discrepancies between the bits file and their corresponding fasm?
Which ones should be trusted?

@litghost
Hmm...I was always looking for the bits in the bits files. And in the archive you sent I also see these three bites across all of the bits files. However it's not always the case for fasm.
How come we have discrepancies between the bits file and their corresponding fasm?
Which ones should be trusted?

Interesting observation.  bits is trustworthy, but the absence of unknown from the  fasm is a signal too.  It may be a bug in bit2fasm, but I suspect not.  Instead, the question becomes which FASM feature matched that bit?  It may provide a hint as to what is going on.  If it is in fact a bug in bit2fasm, we should fix it.

@litghost @mithro I analyzed the problem and it turned out to be a problem in fasm (https://github.com/SymbiFlow/fasm/pull/15). I am not saying this fix is very clean, but it proves that the bits are in fact constant.
I will now focus on accommodating for that in the prjxray flow. I will go with using the CFG_CENTER_MID rather than the synthetic tile.

I analyzed the problem and it turned out to be a problem in fasm (SymbiFlow/fasm#15).

Perfect, thanks for finding that.

The remaining unknown bits are always set to on. In order for the harness to work those bits have been added to the database with the following PR #1046.


Move to new hyperledger-solang org

solang has moved from https://github.com/hyperledger/ to https://github.com/hyperledger-solang/

You can probably remove .github/workflows/dco.yml - the new DCO bot is run by CNCF and is more stable.

@ryjones DCO is preventing merge after I removed .github/workflows/dco.yml

See below:

found it